What is wrong with evidence-based policy making?

Why waiting for scientific evidence before taking decisions means always being behind the curve on risk.

Sign Up

Get the New Statesman's Morning Call email.

The government recently announced it will be forming a Joint Biosecurity Centre, which looks as though it will run along similar lines to the Joint Terrorism Analysis Centre, with experts producing analysis and setting threat or alert levels allowing ministers to make informed policy decisions. This is very welcome.

With hindsight, it seems extraordinary that there was previously no central coordinating point for understanding threats to the nation’s health. But for it to work, the new biosecurity alert level must be based not just on scientific evidence but on the application of sound risk judgements. If you wait for scientific evidence before you take decisions then you will always be behind the curve on risk. 

In counter-terrorism, all threat level judgements are, by definition, not based on scientific evidence. If you knew enough detail about a terrorist plot before it took place, you could stop it. Therefore the threat would not exist. If you waited for the evidence about a terrorist plot then the bomb would already have gone off. That can – of course – happen, and sometimes the threat level goes up after an attack.   

But, at its best, a national threat level for terrorism is a warning system; it aims to make risk-based judgements about the level of threat before there is sufficient evidence by charting the likelihood of an attack happening. This enables action to be taken to mitigate the threat through protective measures, such as greater security checks. In most cases there will be fragments of evidence that indicate a potential threat, meaning the threat level might have to be raised. Very seldom is there enough evidence for the decision to be clear and unambiguous, so there’s a judgement call to be made. In almost all cases the point at which the decision needs to be made will come before the acquisition of the critical data.

Perhaps the problem is that even the phrase “scientific evidence” is unhelpful because it implies complete knowledge. This is seldom true, and is most definitely not so in the case of coronavirus. The word “evidence” in this case carries a false authority. A better phrase might be “what we know from the science at this time”. We should accept that we are not in a position to wait until the scientific evidence is complete and we should be clear that that will likely always be the case. This means there will invariably be a need for a step between the scientific evidence that we have so far and a policy decision, which is the application of human judgement based on understanding and instinct.  

The counter-terrorism threat assessment system has evolved over now nearly 20 years of changing threat, and it is possible to draw out a few key aspects to the process: 

  1. Start with what you do know. Make sure there is an effective method for logging and managing all incoming information even if it is only fragments.
  2. Start counting early. Count everything. The data that enables you to look for patterns and draw conclusions will only be there if you started collecting it before you knew you needed it. 
  3. Prioritise wider understanding. Learn everything possible about the threat and its history.
  4. Watch the threat all the time. As soon it starts to change, look at the patterns and map probable trajectories: what is likely to happen based on general and specific knowledge from previous experience and other cases?  
  5. Have a set of clear criteria for analysts to use as a guide for making changes to the description of the level of risk. 
  6. Take the science out. Apply reason and instinct. Even if something is 90 per cent unlikely, this could be the 10 per cent. Don’t let the science crowd out the judgement.

The final step is the crucial one. There is a point where statistics and probability are genuinely unhelpful. When President Obama was asked to take a decision on whether to go ahead with the raid on Osama bin Laden’s compound, he was given two competing percentages of likelihood that Bin Laden would be there (70 per cent on one, 50 per cent on the other). It held up the decision when time was precious. Neither could have been wholly accurate; he was either there or he wasn’t. We have become used to probability judgements and other statistical calculations as substitutes for certainty, but they aren't, and they can have disproportionate influence on decision-making. 

Nothing is perfect; any system where a critical decision rests on human judgement will not be. There is the temptation to avoid having failed to warn by being overcautious and raising the threat level to cover any possibility that something bad could happen (and knowing that it is unlikely to be possible to show what would have happened if the mitigations hadn’t been imposed). But crying-wolf can very quickly undermine the credibility of the system. 

The public needs to know the level only rises when it needs to. Getting the balance right is tricky.

Of course, the key question is: who takes the risk-based decision? It seems that the Joint Biosecurity Centre will, like the Joint Terrorism Analysis Centre, have an independent analytical function. It will collect and analyse data and will advise the chief medical officers of a change in the Covid-19 alert level. They will then advise ministers.  

An independent body with devolved responsibility for risk judgements will be able to build trust and expertise without the accusation that the alert level is being manipulated for political purposes.  

But as with terrorism, the law and the Bank of England, once decision-making has been devolved to a responsible body, ministers cannot interfere even if they don’t like it. They may find that frustrating. In fact, though, independence also works for the government, which will be able to blame the Joint Biosecurity Centre if it gets it wrong. 

It takes time to build independence and authority, and this is a tall order at the peak of a crisis. It may be that the responsibility is too much for a new organisation to bear. This government has said repeatedly that it has been guided by the scientific evidence. It is to be hoped that in the future it will be guided by the sound risk judgements of the Joint Biosecurity Centre.  

Suzanne Raine worked for 24 years in the Foreign and Commonwealth Office on foreign policy and national security issues. She is an affiliated lecturer in the department of politics and international studies at the University of Cambridge

Free trial CSS