Statistic cited to defend austerity partially based on Excel error

How bad did Reinhart and Rogoff get it?

 

Reinhart and Rogoff

It's always hard to work out how much policy is based on actual evidence, rather than the preconceptions of politicians and policymakers, but if any research has had an effect, it's surely Carmen Reinhart and Ken Rogoff's 2009 book This Time it's Different. It's the source of a claim which has outgrown its roots, and come to be cited in policy debates worldwide: that growth drops precipitously if the ratio of debt to GDP rises above 90 per cent. But now, a new paper shows that that claim is partially the result of some astonishing oversight – including an error in the authors' Excel spreadsheet which excluded five countries from the analysis.

The book itself examines the link between the ratio of debt to GDP and growth rates in a raft of countries from World War II onwards. It finds that the higher the debt to GDP ratio, the lower real growth in those countries – and that there is a massive drop of debt to GDP ratios rise above 90 per cent, when the average growth rate becomes slightly negative.

To be fair to Reinhart and Rogoff (or R&R, as the cool kids do not say), the claim they make has been spun out of proportion by supporters keen to use it for political ends. The authors don't explicitly present the 90 per cent level as a cliff, just highlight what the data says; and they don't draw a causal inference, speaking, as they point out today, "of 'association' and not 'causality.'"

Herndon, Ash and Pollin

Even so, however, no other researcher has been able to replicate their "association", and no satisfactory explanation has been given as to why that is. Until now. The new critique, "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff" by Thomas Herndon, Michael Ash and Robert Pollin (HAP, in economistspeak), is damning. It highlights three inaccuracies in R&R: "coding errors, selective exclusion of available data, and unconventional weighting of summary statistics".

Of those, the first is the most painful, albeit the least important. Reinhart and Rogoff simply added up their spreadsheet wrong. Mike Konczal's report on the paper illustrates the error: the blue box encloses the cells which R&R used to estimate the average; notice how it doesn't go all the way to the bottom? It should:

Missing out the last five rows – particularly Belgium, which had an average growth rate of 2.6 per cent during the years it had a debt to GDP ratio above 90 per cent – changes the average from -0.1 per cent to 0.2 per cent.

That error explains why no-one else could replicate R&R's findings – but the other two problems cast further doubt on whether even the 0.2 per cent figure is acceptable.

The HAP paper finds that R&R exclude certain years in certain countries for no documented reason. These include five years in which New Zealand has a debt to GDP ratio of over 90 per cent. With those years included, the average growth during New Zealand's six years above the threshold is 2.58 per cent; with them excluded it plummets to -7.6 per cent. Similar, albeit smaller, results are found for Australia and Canada, which are also excluded for short periods immediately after the war.

Finally, the HAP paper addresses the way in which R&R weight the results. Each country's data is averaged out, and then the average of those averages is found. That has the effect of valuing the 19 data points that the UK offers above 90 per cent debt/GDP – which average 2.4 per cent growth – with the same weight as the single year that New Zealand offers, when growth was -7.6 per cent.

With all the criticisms applied, the HAP paper reports that the average growth rate for years with a debt/GDP ratio is not -0.1 per cent, but 2.2 per cent. The steep drop-off at 90 per cent disappears; and the credibility of those who cited it should take a hit.

Reinhart and Rogoff Respond

But Reinhart and Rogoff aren't taking it sitting down. With an astonishing turnaround, they have issued a response – published at 3am Boston time – which addresses the critique.

They concede the Excel error – "full stop" – but give a defence for the other two points. The full data for the years excluded was not available when they did their research, they argue, and so while it may make sense to include now, they cannot be held responsible for its absence:

This charge, which permeates through their paper, is one we object to in the strongest terms. The “gaps” are explained by the fact there were still gaps in our public data debt set at the time of this paper.

They also defend the odd choice of weightings, saying that:

Our approach has been followed in many other settings where one does not want to overly weight a small number of countries that may have their own peculiarities.

That is, they argue that just because there is more data for Britain than New Zealand, that does not mean Britain should be weighed more strongly, since that runs the risk that its "peculiarities" might alter the result.

The problem is that neither approach is obviously preferable. While R&R have a point, so to do HAP – which leaves us in the position of questioning the viability of such analysis in the first place.

But R&R make one final defence:

[Herndon et al], too, find lower growth associated with periods when debt is over 90 per cent. Put differently, growth at high debt levels is a little more than half of the growth rate at the lowest levels of debt.

They published this table, via Business Insider, to make the claim clearer:

Does it even matter?

But here's the thing: Reinhart and Rogoff's claim that the HAP paper agrees with them is more evidence of the supreme obviousness of their associative claim. "A high ratio of debt to GDP is correlated with low growth in GDP" is not an interesting finding, it's as close to a mathematical truism as economic statements come. Reinhart and Rogoff's paper is only important insofar as people have read two things into it which aren't true: firstly, that high debt to GDP ratios cause low growth; and secondly, that there is a discontinuity at 90 per cent, where things get much, much worse.

Reinhart and Rogoff themselves disavow the first claim, writing that:

We are very careful in all our papers to speak of "association" and not "causality".

And the second claim has been put to bed by the Herndon et al paper. There is no major drop at 90 per cent, because that was an artefact of incomplete data, errors in coding, and an odd weighting system.

(Incedentally, the 90 per cent discontinuity was a red herring anyway, because it only exists due to the fact that R&R broke up their data into bands 30 percentage points wide. Anyone focusing too heavily on it as a "magic number" simply failed to read the methods section)

And so we are left in much the same place we were beforehand. There remains no evidence that high debt causes GDP growth to slow, rather than slow GDP growth causing high debt. And that lack of evidence will have precisely no effect on public debate, because it's basically all data-free anyway.

There is one change, though. The thesis that Excel is the most dangerous piece software in the world just got a massive boost.

Alex Hern is a technology reporter for the Guardian. He was formerly staff writer at the New Statesman. You should follow Alex on Twitter.

Getty
Show Hide image

Our union backed Brexit, but that doesn't mean scrapping freedom of movement

We can only improve the lives of our members, like those planning stike action at McDonalds, through solidarity.

The campaign to defend and extend free movement – highlighted by the launch of the Labour Campaign for Free Movement this month – is being seen in some circles as a back door strategy to re-run the EU referendum. If that was truly the case, then I don't think Unions like mine (the BFAWU) would be involved, especially as we campaigned to leave the EU ourselves.

In stark contrast to the rhetoric used by many sections of the Leave campaign, our argument wasn’t driven by fear and paranoia about migrant workers. A good number of the BFAWU’s membership is made up of workers not just from the EU, but from all corners of the world. They make a positive contribution to the industry that we represent. These people make a far larger and important contribution to our society and our communities than the wealthy Brexiteers, who sought to do nothing other than de-humanise them, cheered along by a rabid, right-wing press. 

Those who are calling for end to freedom of movement fail to realise that it’s people, rather than land and borders that makes the world we live in. Division works only in the interest of those that want to hold power, control, influence and wealth. Unfortunately, despite a rich history in terms of where division leads us, a good chunk of the UK population still falls for it. We believe that those who live and work here or in other countries should have their skills recognised and enjoy the same rights as those born in that country, including the democratic right to vote. 

Workers born outside of the UK contribute more than £328 million to the UK economy every day. Our NHS depends on their labour in order to keep it running; the leisure and hospitality industries depend on them in order to function; the food industry (including farming to a degree) is often propped up by their work.

The real architects of our misery and hardship reside in Westminster. It is they who introduced legislation designed to allow bosses to act with impunity and pay poverty wages. The only way we can really improve our lives is not as some would have you believe, by blaming other poor workers from other countries, it is through standing together in solidarity. By organising and combining that we become stronger as our fabulous members are showing through their decision to ballot for strike action in McDonalds.

Our members in McDonalds are both born in the UK and outside the UK, and where the bosses have separated groups of workers by pitting certain nationalities against each other, the workers organised have stood together and fought to win change for all, even organising themed social events to welcome each other in the face of the bosses ‘attempts to create divisions in the workplace.

Our union has held the long term view that we should have a planned economy with an ability to own and control the means of production. Our members saw the EU as a gravy train, working in the interests of wealthy elites and industrial scale tax avoidance. They felt that leaving the EU would give the UK the best opportunity to renationalise our key industries and begin a programme of manufacturing on a scale that would allow us to be self-sufficient and independent while enjoying solid trading relationships with other countries. Obviously, a key component in terms of facilitating this is continued freedom of movement.

Many of our members come from communities that voted to leave the EU. They are a reflection of real life that the movers and shakers in both the Leave and Remain campaigns took for granted. We weren’t surprised by the outcome of the EU referendum; after decades of politicians heaping blame on the EU for everything from the shape of fruit to personal hardship, what else could we possibly expect? However, we cannot allow migrant labour to remain as a political football to give succour to the prejudices of the uninformed. Given the same rights and freedoms as UK citizens, foreign workers have the ability to ensure that the UK actually makes a success of Brexit, one that benefits the many, rather than the few.

Ian Hodon is President of the Bakers and Allied Food Workers Union and founding signatory of the Labour Campaign for Free Movement.