Reinhart and Rogoff Exposed

[Surrealistik]

Austrian school/pro-austerity darlings (and PhD Harvard economics professors) Reinhart and Rogoff were exposed as either partisan frauds, bumbling incompetents or both by MIT undergrad Thomas Herndon who reviewed their influential paper, often cited and used to justify policy by fiscal conservatives/proponents of austerity, which falsely suggested that real GDP growth beyond 90% debt to GDP has historically dropped off a cliff, and that an otherwise intensely strong inverse correlation between debt and real GDP growth exists beyond that threshold.

Thomas discovered very basic and egregious Excel spreadsheet coding errors and omissions of important examples directly contrary to their hypothesis and findings (Canada, New Zealand and Australia to name a few) while including non-monetarily sovereign nations (i.e. those that can't issue their own currency) that lacked a free floating currency.

Bloomberg article:

http://www.businessweek.com/articles/2013-04-18/faq-reinhart-rogoff-and-the-excel-error-that-changed-history

More on Herndon:

http://blogs.wsj.com/economics/2013/04/23/the-doctoral-student-who-happed-reinhart-and-rogoff/

A good summation of the drama by economist Mike Norman (excepting the latter-most minute and a half which is about purely political indignation):

http://www.youtube.com/watch?v=TIwNieSmW8I

and, of course, humourist Stephen Colbert:

http://www.youtube.com/watch?v=9-AUKNZL0bM


That all said, while I'm sure it will be difficult, please keep things on point about economics & academics, not politics.

[warty goblin]

Somehow I'd always assumed by the Ph.D. level economists had mastered the same basic software proficiencies most colleges demand of their Stat 101 students. Or probably their Econ 101 students for that matter.

Yet more disturbingly, isn't that whole peer review process supposed to prevent stupid errors getting published? 'Cause for something like this to get published, it takes more than just the authors goofing up...

Once again the delicate butterfly of my optimism gets sucked into the jet engine of human failure.

[Kindablue]

It wasn't published in a peer reviewed journal.

[Surrealistik]

To be honest warty, I'd be highly skeptical that the bogus findings of their study were due exclusively to incompetence; in light of the paper's severe and egregious deficiencies in both methodology _and_ data compilation that two PhD Harvard economists easily should have discerned, an intent to skew for partisan, politically motivated reasons, seems more probable than a litany of obvious and stupid, albeit honest errors.

This theory is, in my view, further reinforced with respect to the question of peer review; it's highly telling that not only was their study _not_ peer reviewed, but Rogoff and Reinhart consistently refused to submit their working data for examination until recently, effectively barring any kind of substantive critique or analysis, which is what lead to their shameful, summary exposure. By this time, a lot of damage on the policy front had already been done (just ask the EU and US Congress).

In summary, it looks like they, beyond the trivial Excel errors:

Weighed the data in a distortive manner,
Did not adequately evaluate the impact of said weighing on results,
Selectively omitted important and in particular, directly contrary (relative to their findings), data,
Did not disclose any of these deficiencies,
Publicly asserted, or at least strongly implied causal links between debt and growth despite lacking proof,
Refused to release their calculations and data for approximately three years,
Seemed to be patronized by Peter Peterson (http://en.wikipedia.org/wiki/Peter_George_Peterson), a notorious fiscal conservative partisan. Further details (http://www.prwatch.org/node/12065).



That all said, the politicians that used their slipshod, unreviewed paper as a basis for austerity policy promotion and formation were grossly and criminally negligent.

[warty goblin]

To be honest warty, I'd be highly skeptical that the bogus findings of their study were due exclusively to incompetence; in light of the paper's severe and egregious deficiencies in both methodology _and_ data compilation that two PhD Harvard economists easily should have discerned, an intent to skew for partisan, politically motivated reasons, seems more probable than a litany of obvious and stupid, albeit honest errors.

The little I've read about this suggests that the Excel error, although stupid, doesn't actually change their numbers all that much. I mostly just find it comical.


This theory is, in my view, further reinforced with respect to the question of peer review; it's highly telling that not only was their study _not_ peer reviewed, but Rogoff and Reinhart consistently refused to submit their working data for examination until recently, effectively barring any kind of substantive critique or analysis, which is what lead to their shameful, summary exposure. By this time, a lot of damage on the policy front had already been done (just ask the EU and US Congress).
The journal wasn't peer reviewed? Why the hell did anybody even pay attention to it in the first place? If you don't get other people to verify that your claims aren't bogus, it really doesn't count for anything.


In summary, it looks like they, beyond the trivial Excel errors:

Weighed the data in a distortive manner,
Did not adequately evaluate the impact of said weighing on results,
Selectively omitted important and in particular, directly contrary (relative to their findings), data,
Did not disclose any of these deficiencies,
Publicly asserted, or at least strongly implied causal links between debt and growth despite lacking proof,
Refused to release their calculations and data for approximately three years,
Seemed to be patronized by Peter Peterson (http://en.wikipedia.org/wiki/Peter_George_Peterson), a notorious fiscal conservative partisan. Further details (http://www.prwatch.org/node/12065).

Weighting data is a hard to figure thing. There are sound methodologies where unequal sample sizes receive equal weights (Type III sums of squares ANOVA for instance) but I don't think I'd be comfortable extending them to data like this. When in observational data land, you want to use survey methodology in my understanding, which would absolutely not weight unequal quantities of information equally. That's just basic stratified sampling design - which is first year grad school stuff.

And there's no way an observational study based on non-random sampling can ever justify a claim of causality, unless you've got a really good scientific theory backing it up. Like, gravity good. For something as complex, and with as many outside sources of variation as economic data, such an inference isn't even on thin ice. It's at the bottom of the lake.


That all said, the politicians that used their slipshod, unreviewed paper as a basis for austerity policy promotion and formation were grossly and criminally negligent.
To be fair to politicians, it's not like they've necessarily got the training to weasel out a BS bit of number-crunching. After all they're chosen via popularity contest, not on the basis of critical thinking and analytic skill. And, at least in America, twenty or thirty odd years of anti-intellectualism has rendered critical thinking fairly unpopular with a lot of voters.

[Surrealistik]

The little I've read about this suggests that the Excel error, although stupid, doesn't actually change their numbers all that much. I mostly just find it comical.

Between the Excel errors and the flaws in methodology, the differences are _substantive_ actually, particularly with a delta of about 2.3%; from -0.1 to ~2.2% real GDP growth. Post critique, the negative correlation between high debt to GDP ratios and GDP growth seemingly endures, though to a _much_ lesser extent.


The journal wasn't peer reviewed? Why the hell did anybody even pay attention to it in the first place? If you don't get other people to verify that your claims aren't bogus, it really doesn't count for anything.

I personally suspect the Harvard name drop + R&R's academic pedigree and partisan pushing of austerity and Austrian/libertarian economics and policy had a lot to do with it. I have little doubt that politics were the primary driver behind its widespread promotion in spite of its lack of peer review and academic rigour.


Weighting data is a hard to figure thing. There are sound methodologies where unequal sample sizes receive equal weights (Type III sums of squares ANOVA for instance) but I don't think I'd be comfortable extending them to data like this. When in observational data land, you want to use survey methodology in my understanding, which would absolutely not weight unequal quantities of information equally. That's just basic stratified sampling design - which is first year grad school stuff.

And there's no way an observational study based on non-random sampling can ever justify a claim of causality, unless you've got a really good scientific theory backing it up. Like, gravity good. For something as complex, and with as many outside sources of variation as economic data, such an inference isn't even on thin ice. It's at the bottom of the lake.


Exactly. What Rogoff and Reinhert did in promoting austerity and declaring (or otherwise minimally strongly insinuating) causality that wasn't in any way even remotely proven or even necessarily suggested by the data was reckless and in my opinion almost criminally irresponsible, given how much weight their words carry as top level academics, and the probable impact of this paper.

Though not nearly an impartial commentator, I feel Krugman summarizes the influence of R&R's paper fairly well here, and the political undercurrents associated with it: http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html?partner=rssnyt&emc=rss&_r=0

Also I found the following article on the subject of R&R's influence particularly interesting:
http://qz.com/75117/how-influential-was-the-study-warning-high-debt-kills-growth/


That said egregious omissions and disingenuous (or at best questionable) weightings only compound their failings.


To be fair to politicians, it's not like they've necessarily got the training to weasel out a BS bit of number-crunching. After all they're chosen via popularity contest, not on the basis of critical thinking and analytic skill. And, at least in America, twenty or thirty odd years of anti-intellectualism has rendered critical thinking fairly unpopular with a lot of voters.

Again, I don't think it has as much to do with anti-intellectualism as it does with politics, libertarian/Austrian school agenda pushing and the institution of corporatist/plutocratic policy, one of the primary goals of which is the massive downsizing of governmental social spending and implementation of austerity. The politicians of this leaning at the ECB and in Congress were obviously more than happy to cite seemingly impervious (from a political, not academic viewpoint) works by Harvard PhDs that were strongly aligned with their own stance regardless of their lack of academic review. Furthermore, again, the study itself was, so far as I can tell, essentially paid for by a prominent austerity advocate.

[valadil]

Somehow I'd always assumed by the Ph.D. level economists had mastered the same basic software proficiencies most colleges demand of their Stat 101 students. Or probably their Econ 101 students for that matter.


In all likelihood, it probably was one of their grad students working Excel for them.