Integrity: What to do if your academic paper is challenged

In the news recently is University of Massachusetts Amherst economics doctoral student Thomas Herndon, who reworked a very influential paper, and found some real calculation errors. He dug into the “Growth in a Time of Debt” paper by Harvard’s  Carmen Reinhart and Kenneth Rogoff as part of a class assignment, where students were asked to rework the numbers of an existing work.

Once the numbers were found to be wonky, Herndon wrote a paper with the assistance of the class’s Professors, Michael Ash and Robert Pollin. The paper, “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff” caused a storm.  The Reinhart and Rogoff paper supported the theory that cutting borrowing (and spending) is the right thing to do in a recession. Right wing politicians have liked brandishing it as evidence over the last while for their Austerity programs. However reworking the numbers shows that it is clear the reverse is true. Turns out that borrowing and spending promotes growth, supporting the standard economic theory that getting out of a recession is about creating demand by helping people earn and spend. 

End result? Herndon and the two co-writer professors are heros, and Reinhart and Rogoff have a lot of egg on their face. They have subsequently admitted at least some of the errors but are still arguing for their point of view. In my opinion their refutation is very strong, and they are quite simply going against the facts.

I have had a similar experience to recount, but the ending was different.

In 1998 I took a class in Corporate Governance at Yale from Professors Paul MacAvoy and Adjunct Professor Ira Millstein. As part of that course a small group had to write a paper on one of a number of set topics, and present to the class. Our group got the topic of challenging the then just published paper: “The Active Board of Directors and Performance of the Large Publicly Traded Corporation“, written by none other than “Millstein, Ira M.; MacAvoy, Paul W“.

It was a tough challenge to take on.

Prof MacAvoy, who is now emeritus, is a leading academic in governance as well as telecommunication and electricity regulation. Millstein was the engine behind Weil Gotshal and Manges (an elite law firm), advisor to many top US boards and also an expert in regulation and anti-trust law. They were pros, and taught the course as a team and had done so for many years. Turns out that the course we took was the heavy basis for the first version of the OECD Principles of Corporate Governance, authored by Millstein.

Each of these gentlemen have ridiculous resumes, formed from a lifetime of achievement and integrity. So when we took on the challenge we knew we were up against it.

I read the paper, which compared the corporate performance measured in ‘economic profit’ against the quality of boards, measured by CalPERS, a huge fund, who provided ABC type ratings of boards of directors for top US companies.

I dug into the data, and then, with interest piqued, kept digging, acquiring the original spreadsheet, working through the formulae and the original macros and so on. The spreadsheet would suck in all of the financial data for a company, calculate the ‘economic profit’ between two periods, record it and then move to the next company. After that it was a simple regression against the CalPERS data. The Millstein/MacAvoy paper had found that there was a significant positive correlation between the quality of a board and the change in economic profit.

I could not get the economic profit model to work, so I ran the same regression using other measures, such as total return to shareholders and revenue growth over the same period. And I struck a problem. These numbers showed almost no correlation, and even negative correlation on some measures. We checked it, bundled it into a paper and presentation and presented that back to the Professors and class.

Now let’s pause. If you were the Professors, what would your reaction be?

MacAvoy and Millstein’s reaction was amazing. Not only did they accept that the findings could be valid, but Prof. Paul MacAvoy almost immediately afterwards asked whether I was interested in staying on after the academic year had finished, which it was about to, to dig into the issue some more to write a paper.

Instead I went on a lengthy motorcycle trip, but 8 months or so later called in from Singapore, asking whether the offer was still open. It was and I was offered a very generous deal. Incidentally I was funded by the rather right wing Olin Foundation, and paid by Yale under the same program that they use to pay elite academics and sports people doing very short appearances.

I worked first with the original research assistant and we got the model working again, and then I explored. Over time I realised that there were errors in the original model and how it was run. I fixed it and worked it through again, and using several metrics rather than just economic profit. From there I wrote a Yale Working Paper, “Corporate Performance and Corporate Governance”. The paper to my mind showed that the MacAvoy/Millstein paper was flawed, and that there really was no correlation proved. However they were willing to accept the that their own work was valid as my paper still showed a small, not really significant, positive correlation. There was no doubt in my mind that they would have withdrawn the paper (or whatever one does) if the results were shown to be wrong.

In the Reinhart and Rogoff case their paper was proven to be quite wrong, yet they stood behind their results regardless. In this case MacAvoy and Millstein encouraged and even paid for my challenge, and were able to make a well informed decision on how to proceed. 

To me the stories show three things. 

Firstly, the power of integrity. MacAvoy and Millstein knew that an error-filled paper was simply inexcusable, and were willing not only to ask people from their class to challenge it, but then to invest funding dollars and substantial time in helping to firm up that challenge. Tenure helps, but integrity was what drove this behaviour.

Secondly, the power of checking the numbers. MacAvoy and Millstein were completely reliant on the very smart but still solo operating research assistant. The RA, who turned out to be my next door neighbour, created the spreadsheet, ran the numbers and did all of the the statistical work – all of which I had to replicate. I would not expect the aging professors to drive a spreadsheet like that, but I would recommend that the person performing the analysis is checked by someone else, or works with a deeply analytical partner.

Thirdly, we should devalue or even ignore any research where we are not able to see the original data and the models used to run the numbers, and have the ability to re-run the exercise. These should be posted in open formats, be very easy to download and re-run and be accessible by the public without charge. Indeed why are papers not presented with the working model in electronic form embedded in the paper? The move to Open Research will help significantly here, and any funder should be mandating publishing not (just) in peer reviewed journals, but in an online and open way.