Political middlemen and dart-throwing chimps

Martha Gill's "Irrational Animals" column.

Predicting the weather was once quite an interesting profession, needing skill in reading the instruments, intuition in deciphering the skies and years of experience in putting it all together. Now it’s the kind of job Nick Cage’s character would be given in a heavy-handed satire of the American dream, possibly also starring Michael Caine. We don’t need these skilled individuals any more – computers do all that. We just need an algorithm and a mouthpiece.

And so to Nate Silver – one of the biggest winners of the US presidential election. As the race neared its end, becoming “too close to call”, with money and opinions frantically changing hands, the New York Times blogger was calmly and correctly predicting voter outcome in every single state. He had what others didn’t – a formula to convert polling information into probabilities – and it turned out to be dead-on. He was not alone in getting it right but he was among the few. Many failed spectacularly.

Here’s Newt Gingrich on Fox News on 25 October: “I believe the minimum result will be 53-47 [per cent] Romney, over 300 electoral votes, and the Republicans will pick up the Senate. I base that . . . on just years and years of experience.” And here’s the GOP strategist Karl Rove in the Wall Street Journal on 31 October: “It comes down to numbers. And in the final days of this presidential race, from polling data to early voting, they favour Mitt Romney.”

These were not small errors. These people were standing in pre-hurricane wind and predicting sunshine. Are pundits more often wrong than not, or was it just this particular election that threw them? And how often do the statistics spewed out by experts hit the mark? One study found a statistic for it.

Algorithm blues

In the 1980s, a psychologist called Philip Tetlock took a group of journalists, foreign policy experts and economists – 284 of them – and spent the next two decades bombarding them with questions: would the dotcom bubble burst? Would George Bush be re-elected? How would apartheid end?

After analysing 82,361 predictions, Tetlock found that his experts performed worse than random chance. In short, they could have been beaten by dart-throwing chimps.

The reason was confidence. Tetlock found that the more often pundits appeared on TV, the more likely they were to be wrong. Their strong opinions were causing them to ignore dissenting facts or explain them away, leaving them trapped, he said, in the cage of their preconceptions.

Now, semi-expert middlemen are being squeezed out as the focus shifts to minute data analysis. Silver is one of the winners of this change but on the losing side is a whole industry of political forecasters. And it’s not just true of politics. Finance has been moving that way for a while. In UBS’s recent swath of job cuts, at least one trader, David Gallers, was replaced with an algorithm.

Difficult times for the old school, but what of the new? Silver expressed his concerns to the Wall Street Journal: “You don’t want to influence the system you are trying to forecast.” Only one problem with the new machines, then – accuracy. They’re so good that they might start controlling the weather.

Newt Gingrich opining away on Fox News. Photograph: Getty Images

Martha Gill writes the weekly Irrational Animals column. You can follow her on Twitter here: @Martha_Gill.

This article first appeared in the 19 November 2012 issue of the New Statesman, The plot against the BBC

DebateTech
Show Hide image

Politicians: it's no longer OK to know nothing about technology

It’s bad enough to joke about not being "techy"; it's worse to back a piece of legislation from a position of ignorance. 

Earlier this week, facing down a 600-strong battalion of London’s tech sector at a mayoral hustings in Stratford, Zac Goldsmith opened his five minute pitch with his characteristic charm. “I’m not very techy!” he exclaimed. “I understand coding about as well as Swahili!”

Pointless jibe at a foreign language aside, this was an ill-chosen way to begin his address - especially considering that the rest of his speech showed he was reasonably well-briefed on the problems facing the sector, and the solutions (including improving broadband speeds and devolving skills budgets) which could help.

But the offhand reference to his own ignorance, and the implication that it would be seen as attractive by this particular audience, implies that Goldsmith, and other politicians like him, haven’t moved on since the 90s. The comment seemed designed to say: “Oh, I don't know about that - I'll leave it to the geeks like you!"

This is bad enough from a mayoral hopeful.  But on the same day, the Intelligence and Security Committee of Parliament filed its report on the Draft Investigatory Powers Bill, the legislation drafted by the Home Office which will define how and how far the government and secret services can pry into our digital communications. Throughout, there's the sense that the ISC doesn't think the department behind the bill had a firm grasp on the issues at hand. Words like "inconsistent" and "lacking in clarity" pop up again and again. In one section, the authors note:

"While the issues under consideration are undoubtedly complex, we are nevertheless concerned that thus far the Government has missed the opportunity to provide the clarity and assurance which is badly needed."

The report joins criticism from other directions, including those raised by Internet Service Providers last year, that the bill's writers didn't appear to know much about digital communications at all, much less the issues surrounding encryption of personal messages.

One good example: the bill calls for the collection of "internet connection records", the digital equivalent of phone call records, which show the domains visited by internet users but not their content. But it turns out these records don't exist in this form: the bill actually invented both the phrase and the concept. As one provider commented at the time, anyone in favour of their collection "do not understand how the Internet works". 

Politicians have a long and colourful history of taking on topics - even ministerial posts - in fields they know little to nothing about. This, in itself, is a problem. But politicians themselves are often the people extolling importance of technology, especially to the British economy - which makes their own lack of knowledge particularly grating. No politician would feel comfortable admitting a lack of knowledge, on, say, economics. I can’t imagine Goldsmith guffawing "Oh, the deficit?  That's all Greek to me!"  over dinner with Cameron. 

The mayoral candidates on stage at the DebateTech hustings this week were eager to agree that tech is London’s fastest growing industry, but could do little more than bleat the words “tech hub” with fear in their eyes that someone might ask them what exactly that meant. (A notable exception was Green candidate Sian Berry, who has actually worked for a tech start-up.) It was telling that all were particularly keen on improving internet speeds -  probably because this is something they do have day-to-day engagement with. Just don't ask them how to go about doing it.

The existence of organisations like Tech London Advocates, the industry group which co-organised the hustings, is important, and can go some way towards educating the future mayor on the issues the industry faces. But the technology and information sectors have been responsible for 30 per cent of job growth in the capital since 2009 - we can't afford to have a mayor who blanches at the mention of code. 

If we’re to believe the politicians themselves, with all their talk of coding camps and skills incubators and teaching the elderly to email, we need a political sphere where boasting that you're not "techy" isn’t cool or funny - it’s just kind of embarrassing. 

Barbara Speed is a technology and digital culture writer at the New Statesman and a staff writer at CityMetric.