Support 100 years of independent journalism.

  1. Science & Tech
16 February 2021updated 17 Feb 2021 9:14am

Why the trial of Covid-19 drug tocilizumab is a model of good science

Unlike the 84 coronavirus-related scientific papers that have now been retracted, the tocilizumab study is transparent and plausible.

By Stuart Ritchie

Pandemic science is a rollercoaster. Almost every week, there’s a new first-rate piece of research that helps make serious progress in the fight against Covid-19. At the same time, there are usually clear examples of how not to do science. Last week was no exception.

Our dose of good science came in the form of the immunosuppressive drug tocilizumab. The Prime Minister had some trouble pronouncing its name at a press conference, and I don’t blame him. But however you say it, Oxford’s Recovery trial has found it seems to work.

The new study hasn’t yet been peer-reviewed – it’s been posted online as a preprint, the preliminary kind of paper released for discussion prior to (or during) review. It reports a relatively small but still important effect: a 14 per cent reduction in the risk of death from Covid-19 for patients who were given tocilizumab (along with the known benefit of steroids, which most of the patients in the trial were also taking). Based on this finding and a previous, smaller study, tocilizumab is now available to coronavirus patients on the NHS. 

That’s the good news. Here’s the bad: the website Retraction Watch last week announced that its list of retracted coronavirus-related scientific papers has now added its 84th entry. The paper in question was on Covid and obesity, and was withdrawn from the scientific literature due to:

“inadvertent errors… that unfortunately passed unnoticed during the extremely rapid review and publication process at the peak of the Covid-19 pandemic”.

Sign up for The New Statesman’s newsletters Tick the boxes of the newsletters you would like to receive. Quick and essential guide to domestic and global politics from the New Statesman's politics team. The best of the New Statesman, delivered to your inbox every weekday morning. The New Statesman’s global affairs newsletter, every Monday and Friday. A handy, three-minute glance at the week ahead in companies, markets, regulation and investment, landing in your inbox every Monday morning. Our weekly culture newsletter – from books and art to pop culture and memes – sent every Friday. A weekly round-up of some of the best articles featured in the most recent issue of the New Statesman, sent each Saturday. A weekly dig into the New Statesman’s archive of over 100 years of stellar and influential journalism, sent each Wednesday. Sign up to receive information regarding NS events, subscription offers & product updates.
I consent to New Statesman Media Group collecting my details provided via this form in accordance with the Privacy Policy

Other papers on the Retraction Watch list have been pulled due to dodgy data, dodgy peer-review and plagiarism; some were retracted for unknown reasons

So why am I so optimistic about the tocilizumab study? Why am I betting it won’t go the same way as the inglorious 84? What should we be looking for that distinguishes good and bad research on Covid – or, indeed, on anything else?

The first point is transparency. The Recovery team has been admirably open about how its data would be analysed. Although every clinical trial has to be registered, it’s still depressingly common for researchers to be overly flexible: changing the measurements they’re focusing on, shifting to a different type of statistical test if they get disappointing results, or splitting the data up into subgroups on a whim. All this makes results less reliable, since the more analyses you run, the more likely your finding is to be a fluke.

But the Recovery trial had an analysis plan, written and posted online before anyone saw the data. The researchers stuck to it, and whenever they wanted to run an analysis they hadn’t planned, they simply said so (several times in the preprint, they note they ran an “exploratory” analysis, and the reader can thus take the results with a pinch of salt). Being open about the often messy process of doing research is refreshing in a world where most scientific papers are written as if everything were planned in advance, but where that’s hardly ever true.

I mentioned that tocilizumab has a relatively small effect – as the researchers put it, on average you have to treat 25 patients with the drug to prevent one death. But that small effect is, perhaps paradoxically, our second reason to be optimistic. Very large effects aren’t particularly common in medical treatments (for drugs, at least… let’s leave aside the obvious large effect that “surgery to remove a bullet lodged near an artery” would have). If the effect had been apparently dramatic, it would have raised suspicions, since such outcomes are usually too good to be true.

That kind of suspicion is currently being levelled at a new study of Vitamin D for Covid-19, also recently released as a preprint. I’ve previously written about the ambiguous evidence suggesting Vitamin D can help combat the virus, but this preprint appeared to be the clincher: a randomised controlled trial showing, as the MP David Davis explained in a viral tweet, an 80 per cent reduction in intensive care visits and a 60 per cent reduction in deaths.

Sadly, the study was not, in fact, a randomised controlled trial (the authors have now acknowledged this, despite what they previously wrote in their preprint). There are also several problems with its methodology. But the main results themselves should have had us on guard: it’s a priori implausible that a common vitamin would cut Covid-19 deaths by well over half.

That doesn’t rule out a smaller effect of Vitamin D, and we might find one as future trials – trials that are actually randomised and controlled – report back. But a 60 per cent reduction in deaths? Come on, man.

Its transparency and its plausibility are just two reasons to feel positive about the Recovery trial. There are other reasons, such as its large sample size, its solid randomised design and the good, theoretical grounds to expect an anti-inflammatory drug like tocilizumab to help with a disease such as Covid-19. You won’t find many of these advantages in the now-retracted studies or in the Vitamin D “trial”.

But let’s look on the bright side. The contrast between the highs and lows of Covid-19 research is a good lesson in differentiating solid from shoddy science.

[see also: Does vitamin D really help combat Covid-19?]