The risks of imprecision

When being half-right can be worse than being wrong.

Chris Dillow has a nice post up, "in praise of imprecision". He argues that, in far too many situations, we argue over tiny differences in estimates when the overall answer is basically known. What's GDP growth for last year? It's basically flat. Yet for all the arguments, you would think that the difference between -0.3 and -0.1 per cent – or even between -0.1 and 0.1 – was the difference between life and death.

He illustrates this with a few neat little guesstimates. For instance:

How much does welfare scrounging cost the economy? Guesstimate the number of scroungers. Guesstimate the value-added they'd contribute if they were working. Express as a proportion of GDP. For plausible values, it's a small number.

Or:

What impact will the small uprating in the minimum wage have on jobs? The adult rate will rise by 1.9%. Economists forecast inflation this year of 2.5%, so this is roughly a 0.6% real fall. Let's call the price-elasticity of demand for labour 1.5. The Low Pay Commission estimates (pdf) that 5.3% of jobs are around minimum wage ones. Multiply these three numbers together and we get 0.048%. Multiply by the number of jobs in the economy (29.73m) and we have roughly 14,000.That's roughly one-eleventh of the sampling variability of employment figures.

It's worth pointing out that the same idea has been applied pretty consistently to the claim that families with three generations of worklessness are a public policy problem. We don't know how many there are – and nor does the government, we now know – but study after study has suggested that the number is tiny.

There are only 15,000 households with two generations which have never worked, and a third of them are because the younger generation left full time education within the last year. On top of that, less than 1 per cent of young people have never worked by the age of 29, so the younger generation is normally the one most likely to pull a family out of worklessness. Whatever the number is, in other words, it's really, really small.

But it's important to note the downside to imprecision. The way common knowledge is disproved is rarely through wholesale upheaval. Instead, it's a gradual process of refinement: new estimates are put out, slightly lower than the old ones; then lower estimates still; and they get steadily lower, until suddenly you realise that the conventional wisdom was wrong.

It's a lot harder to turn an estimate of "recession" into an estimate of "growth" through gradual refinement than it is to turn an estimate of "-0.3 per cent" into one of "0.5 per cent". So there's more of a danger that we'll be stuck with half-truths.

But with that danger in mind, the absence of accepted imprecision is still keenly felt in Whitehall. Too frequently, "no statistics" is used to imply "we have no idea of the magnitude of this problem" – but that's not true. We actually know quite a lot, albeit imprecisely. The trick is acting on it.

Photograph: Getty Images

Alex Hern is a technology reporter for the Guardian. He was formerly staff writer at the New Statesman. You should follow Alex on Twitter.

Getty Images.
Show Hide image

Why Labour's manifesto wasn't regressive

The Institute for Fiscal Studies' analysis did not take into account the progressive effect that most of the party's policies would have. 

Think tankers, for example at the Institute for Fiscal Studies, the Resolution Foundation and IPPR, often like to use ‘distributional analysis’ to assess the impacts of policy on households, and these analyses are often picked up on in wider debates over fairness.

To test whether a policy change is ‘progressive’ – where its impact on poorer households is more positive than it is on richer households – a preferred method is to group all households in the population into buckets, ordered from lowest income to highest, and show the average effect of a policy for each bucket.

The benefits of this type of analysis are obvious. The relatively complicated question of progressivity can, at least for a given metric, be reduced to a visually clear and accessible chart: answering the question of fairness seemingly irrefutably, at least in quantifiable terms.

An influential part of the IFS’s excellent election manifesto analysis was just such a chart, showing the distributional impact of all tax and benefit proposals from the Conservative, Labour and Liberal Democrat manifestos (reproduced below).

The analysis is striking on two accounts. First, it appears to show that the policies in Labour’s manifesto are almost perfectly regressive: from the second to the ninth decile, the poorer a family is, the worse off Labour’s plans would make them. Second, Labour’s plans appear pretty regressive even relative to the other two parties: almost as regressive as the Conservatives, and far more so than the Liberal Democrats.

This chart in particular has helped fuel a broad, alternative narrative that has emerged about the Labour manifesto since the election. This narrative suggests that, far from being radical, the impact of the policies recommended wasn’t even redistributive. For example, John Rentoul reproduced the same chart in his article for the Independent and Andrew Harrop leant on IFS analysis for his piece in the Guardian. Less formally, the arguments have attracted particular traction on social media with commentators such as Robert Peston at ITV and Jeremy Cliffe at the Economist reposting the chart on Twitter.

***

Is this correct? Would the effects of implementing Labour’s manifesto be not only to take away from the poor, but to take away far more than they would from the rich? The answer is either an unequivocal ‘no’ or ‘we don’t know because the analysis hasn’t been done’, depending on the interpretation of ‘effect’. But the answer certainly is not ‘yes’.

Commentators have misinterpreted the IFS work in two important ways.

First, the IFS did not assess the whole of the Labour manifesto. They did not even reflect most of it.

Distributional analysis is most effective in assessing impacts that can be measured, reasonably unambiguously, in a monetised form. Quite sensibly then, and as is often standard practice, the IFS applied their analysis to tax and benefit reforms only – excluding services or other government programmes. Even tax measures where the ‘effective incidence’ (the final, often indirect impact of a tax on households, as opposed to the entity that might have paid the tax in the first instance such as a firm) is equivocal, such as corporation tax, were excluded.

This means the analysis only assessed a fraction of the Labour manifesto’s tax rises (around £8bn from £49bn) and spending commitments (around £4bn from £49bn). In the case of Labour, unlike the manifesto itself which was supposedly cost neutral, the analysis included almost twice as many ‘takeaways’ (tax rises) as it did ‘giveaways’ (additional welfare support).

The final picture, then, is not a reflection of the whole manifesto, but of less than 8 per cent of spending commitments and 16 per cent of tax rises. Some of the measures excluded would likely have had a broadly progressive impact, such as increasing the living wage to £10 per hour, applying VAT to private school fees, scrapping planned giveaways in the taxation of inheritance and capital gains, restoring the Educational Maintenance Allowance and boosting Sure Start.

For those reforms that reflected a stronger principle of universalism – such as the National Education Service, increased child care and scrapping university tuition fees – the effects are less clear and possibly regressive in a strict accountancy sense.

But the IFS work cannot help us reach a conclusion either way because they are excluded from the analysis. Since the UK is one of only six countries among 35 OECD members where the state spends more on welfare in kind than it does in cash, it would be an especially poor outcome for our political discourse if benefits in kind were to be excluded from our redistributive conversation altogether.

The second reason is that the IFS did not at any stage assess manifesto commitments in isolation.

All distributional analysis is essentially an exercise in simulating a number of scenarios, one of which is treated as a ‘baseline’, and all the others as counterfactuals. The estimated impact of a specific counterfactual scenario is essentially just the total difference observed with the baseline scenario.

In addition to the issue of which, and how many, manifesto measures are included, the issue of what is in or out of the baseline relative to the counterfactuals is therefore also critical.

In the case of the recent IFS work, the baseline excluded the suite of policies that have already been legislated for and in some cases already begun to be implemented but nonetheless not come into full effect. The IFS make this clear by labelling each manifesto scenario in the chart as the sum of both ‘current plans’ and the personal tax and benefit announcements from a respective manifesto.  Most notable among them is the introduction of the coalition government’s flagship welfare reform: Universal Credit (UC).

This choice is not without merit but it is also not beyond question: for example the Office for Budget Responsibility’s standard baseline always includes all planned policy that is presently legislated for.

For the purposes of assessing distributional impact, then, the IFS excluded the entire system of UC (which has been legislated for since 2013) from the baseline. This meant that each of the counterfactual scenarios  (including the Labour one) included the full effects of UC as well.

This had an especially large impact on the analysis since the backlog of government reforms currently still being implemented dwarfs the comparatively smaller tax and benefit measures proposed in the manifestos. The Liberal Democrats are the closest to an exception where, unlike Labour and the Conservatives, a far greater proportion of their spending plans affected welfare spending in general, and spending on UC in particular.

Whether this is the right way to conduct analysis depends entirely on the question that is being asked.

If the question is: compared with the world as it is configured today, would the tax and benefit system be more regressive in five years’ time after taking account of all the government’s current reforms that are in the process of implementation in addition to party manifesto pledges? The answer is the same for all parties because of the sheer scale of government reforms in the pipeline: ‘yes’. (Although the point remains that this would still only represent a small portion of total tax and spending announcements from Labour’s manifesto in particular, so the full picture is still unknown).

But if the question was: after taking into account the government’s current reforms to today’s tax and benefit system would the additional impact of the personal tax and benefit reforms scored from Labour’s election manifesto, taken in isolation, be regressive? The answer is absolutely not.

If interpreted correctly, the IFS analysis actually gives the answer to both of these questions. Compared with the world as it currently is (essentially the x-axis itself) all the manifesto policies are broadly regressive, albeit to varying degrees. But compared with the world as currently planned (the IFS’s scenario labelled in green) both the Labour and Liberal Democrat lines are clearly more progressive.

What determined whether the absolute figures presented in the chart were largely negative rather than positive – and therefore perhaps the immediate impression that the analysis leaves with readers – was the choice of baseline. If the ‘current plans’ scenario were to have been the baseline (as is more consistent with the OBR’s standard baseline) the Labour and Liberal Democratic manifestos would have been presented as having a net positive and progressive impact on households in the bottom half of the distribution.

Had that chart been produced instead, any alternative narratives about the Labour manifesto would have been less likely to misinterpret the evidence – although the chart itself would have been no more right or wrong for it.

***

This discussion does not amount to a criticism of either distributional analysis of personal tax and benefits in general, nor of the IFS’s work in particular. The former is an essential tool – when applied appropriately – for assessing particular measures of fairness. The latter execute their work extremely well and in a way that often enhances the quality of the UK’s political conversation.

In particular, on both the main points raised above (the limited number of manifesto measures included and the choice of baselines), the IFS are entirely transparent. The title of their chart is labelled as analysis of ‘personal tax and benefit measures’ only, and the full list of policies included are published for all to see. They also make clear that their analysis has excluded ‘current plans’ from the baseline, and that current plans are included in each counterfactual scenario. The author was also extremely helpful and obliging in answering any queries.

The very worst that can be said of the IFS in this instance is that they have not gone further out of their way to tackle what has grown into – at times – a dangerous misinterpretation of their analysis. But in a political climate where both wilful and accidental misinterpretation of economic evidence and theory is endemic – and often on far more serious issues than the technical progressivity of a single manifesto – their lack of intervention is unfortunate but understandable.

Neither do the points raised here absolve the Labour manifesto from an alternative or progressive critique: in particular, the absence of a more serious reversal of the government’s welfare plans is highly conspicuous, not least given that the Liberal Democrats were able to go far further in their own commitments. And the critique of universalism from a principle of reciprocity, though not the final word, remains an area of useful discussion.

Probably the best conclusion that can be drawn from all this echoes that made by Torsten Bell of the Resolution Foundation in May. Commentators should be careful to not overstate the decisiveness of broad-brush analysis during an election campaign, where seemingly mere technicalities over methods and assumptions can actually shape entirely the final interpretation as much as the facts themselves.

Instead, more attention should be given to the broader direction of travel and the choices on offer.

At the last election, the real debate centred on the size and scale of the state, the balance of state support between welfare in cash or in kind, the divide between young and old, between the super-rich and the rest, and how principles of universalism, reciprocity and targeted redistribution should be brought to bear on questions of fairness. 

These are the tests against which commentators should judge our political parties at the next election as well.