Oli Scarff/Getty Images
Show Hide image

Harvesting the wisdom of the crowd

The “superforecasters” who show remarkable proficiency in answering big geopolitical questions - despite their lack of knowledge.

Can the wisdom of crowds make more accurate predictions than top intelligence experts? Philip E Tetlock, a University of Pennsylvania political scientist, set up the Good Judgment Project in 2011 in order to find out. Funded by the US government, the programme identified a group of “superforecasters” who, despite possessing no particular expertise nor access to any secret information, have shown remarkable proficiency in answering big geopolitical questions – such as whether Serbia would be granted European Union candidacy by the end of 2011, or whether Bashar al-Assad’s regime in Syria would survive in 2012.

Supers, as they are known, are those in the top 2 per cent of participants. Tetlock’s thesis is that by aggregating the guesses of supers, he can derive a probability that comes close to the truth. Although no one can know everything about a given topic, everyone knows a little about something. Tetlock argues that aggregation amplifies what people get right and cancels out what they get wrong.

It sounds too good to be true: a relatively simple approach that overturns decades of received wisdom, a little like the case studies in Freakonomics or Moneyball, but specific to intelligence. What is astonishing is that the approach has been shown to work, with participants achieving accuracy scores well above random chance and supposedly even beating CIA analysts (though the agency apparently refuses to confirm this).

In October, the supers held a public conference at the London School of Economics. Before it opened, I met Andreas, an Italian super who lives in London and works in finance. When I asked him which forecast he was most proud of, he told me that he had closely predicted the number of banks that would fail the EU “stress test” in 2014. But was this forecast his “best”? Rather than giving a quick answer, Andreas instinctively began to deconstruct my rephrased question, asking rhetorically what might be meant by “best”. According to Tetlock, the ability to break down questions is a key skill in becoming a super.

The methods used to collate judgements have varied since 2011 but one of the most successful has been to use a market, or “forecasting tournament”, which works much like a real stock market. Each super is given 50,000 units of a fictional currency known as “Inklings” to invest in various outcomes. If, say, you had bet on Assad surviving a year ago and the odds were then at 10 per cent, you would have won a big return.

What was clear as I sat in the conference room – listening to the likes of the former Ministry of Defence analyst Nick Hare speaking about forecasting defence issues, or the conference organiser Michael Story bemoaning the predictions of media pundits – was that I was surrounded by very smart people. Pablo, the super next to me, was the dean of MBA students at a major business school. One questioner identified himself as working for the Bank of England. Contrary to the promotional pitch of Tetlock’s recent book, Superforecasting: the Art and Science of Prediction, these are not, for the most part, ordinary people, but highly qualified, well-connected and well-read individuals.

One participant suggested that the questions the supers performed worst on were those in their own professional field. When it came to breaking down a question and engaging with it objectively, he claimed, he was unable to see the wood for the trees. Chris, a strategy management consultant who joined the programme in its second year, described this quality positively as “conscious incompetence” – in essence, having an awareness of what you do not know and being open to having your forecasts challenged by others so that together you can arrive at better judgements.

Wanda, a Spanish-speaking guest from Texas (something of a rarity at the conference: the demography of supers seems to be overwhelmingly white and male), told me that she had performed best on questions relating to Iran. To do so, however, she had to be dispassionate, something she found difficult because she has strong opinions about Barack Obama’s recent nuclear deal with Iran. Ironically, for a programme that focuses on contentious political issues, the one taboo among supers seems to be personal politics. When I asked another participant if he would discuss his voting patterns with his super colleagues, he said that to do so would hinder co-operation.

So, what are the chances that this method of intelligence gathering will be taken seriously beyond this select group? I’m not so sure, but I know the right people to ask.

This article first appeared in the 29 October 2015 issue of the New Statesman, Israel: the Third Intifada?

Photo: Getty
Show Hide image

Moss Side Public Laundry, 1979

A new poem by Pippa Little.

Childless I arrive with a rucksack,
own no Silver Cross steered topple-high
by the bare-legged women in check coats
and bulging shoes who load and unload
ropes of wet sheets, wring them out
to rams’ horns while heat-slap of steam
dries to tinsel in our hair, frizzles our lips
gritty with Daz sherbert dabs and the mangle,
wide as a room-size remnant, never stops groaning
one slip and you’re done for…

In the boom and echo of it, their calls swoop
over Cross-your-Hearts, Man. City socks,
crimplene pinks and snagged underskirts,
Maggie Maggie Maggie Out Out Out! blasts
from across the park, whole streets
get knocked out like teeth,
in a back alley on the way a man
jumped me, shocked as I was
by the fuck off! I didn’t know was in me

but which I try out now to make them laugh, these women
who scrub blood and beer and come
with red-brick soap, quick-starch a party dress
while dryers flop and roar
before their kids fly out of school,
flock outside for a smoke’s sweet rest
from the future bearing down of four walls and one man.

Pippa Little’s collection Overwintering (Carcanet) was shortlisted for the Seamus Heaney Centre Award. Her new book, Twist, was published in March by Arc. 

This article first appeared in the 20 July 2017 issue of the New Statesman, The new world disorder