Political middlemen and dart-throwing chimps

Martha Gill's "Irrational Animals" column.

Predicting the weather was once quite an interesting profession, needing skill in reading the instruments, intuition in deciphering the skies and years of experience in putting it all together. Now it’s the kind of job Nick Cage’s character would be given in a heavy-handed satire of the American dream, possibly also starring Michael Caine. We don’t need these skilled individuals any more – computers do all that. We just need an algorithm and a mouthpiece.

And so to Nate Silver – one of the biggest winners of the US presidential election. As the race neared its end, becoming “too close to call”, with money and opinions frantically changing hands, the New York Times blogger was calmly and correctly predicting voter outcome in every single state. He had what others didn’t – a formula to convert polling information into probabilities – and it turned out to be dead-on. He was not alone in getting it right but he was among the few. Many failed spectacularly.

Here’s Newt Gingrich on Fox News on 25 October: “I believe the minimum result will be 53-47 [per cent] Romney, over 300 electoral votes, and the Republicans will pick up the Senate. I base that . . . on just years and years of experience.” And here’s the GOP strategist Karl Rove in the Wall Street Journal on 31 October: “It comes down to numbers. And in the final days of this presidential race, from polling data to early voting, they favour Mitt Romney.”

These were not small errors. These people were standing in pre-hurricane wind and predicting sunshine. Are pundits more often wrong than not, or was it just this particular election that threw them? And how often do the statistics spewed out by experts hit the mark? One study found a statistic for it.

Algorithm blues

In the 1980s, a psychologist called Philip Tetlock took a group of journalists, foreign policy experts and economists – 284 of them – and spent the next two decades bombarding them with questions: would the dotcom bubble burst? Would George Bush be re-elected? How would apartheid end?

After analysing 82,361 predictions, Tetlock found that his experts performed worse than random chance. In short, they could have been beaten by dart-throwing chimps.

The reason was confidence. Tetlock found that the more often pundits appeared on TV, the more likely they were to be wrong. Their strong opinions were causing them to ignore dissenting facts or explain them away, leaving them trapped, he said, in the cage of their preconceptions.

Now, semi-expert middlemen are being squeezed out as the focus shifts to minute data analysis. Silver is one of the winners of this change but on the losing side is a whole industry of political forecasters. And it’s not just true of politics. Finance has been moving that way for a while. In UBS’s recent swath of job cuts, at least one trader, David Gallers, was replaced with an algorithm.

Difficult times for the old school, but what of the new? Silver expressed his concerns to the Wall Street Journal: “You don’t want to influence the system you are trying to forecast.” Only one problem with the new machines, then – accuracy. They’re so good that they might start controlling the weather.

Newt Gingrich opining away on Fox News. Photograph: Getty Images

Martha Gill writes the weekly Irrational Animals column. You can follow her on Twitter here: @Martha_Gill.

This article first appeared in the 19 November 2012 issue of the New Statesman, The plot against the BBC

Show Hide image

7 problems with the Snooper’s Charter, according to the experts

In short: it was written by people who "do not know how the internet works".

A group of representatives from the UK Internet Service Provider’s Association (ISPA) headed to the Home Office on Tuesday to point out a long list of problems they had with the proposed Investigatory Powers Bill (that’s Snooper’s Charter to you and me). Below are simplified summaries of their main points, taken from the written evidence submitted by Adrian Kennard, of Andrews and Arnold, a small ISP, to the department after the meeting. 

The crucial thing to note is that these people know what they're talking about - the run the providers which would need to completely change their practices to comply with the bill if it passed into law. And their objections aren't based on cost or fiddliness - they're about how unworkable many of the bill's stipulations actually are. 

1. The types of records the government wants collected aren’t that useful

The IP Bill places a lot of emphasis on “Internet Connection Records”; i.e. a list of domains you’ve visited, but not the specific pages visited or messages sent.

But in an age of apps and social media, where we view vast amounts of information through single domains like Twitter or Facebook, this information might not even help investigators much, as connections can last for days, or even months. Kennard gives the example of a missing girl, used as a hypothetical case by the security services to argue for greater powers:

 "If the mobile provider was even able to tell that she had used twitter at all (which is not as easy as it sounds), it would show that the phone had been connected to twitter 24 hours a day, and probably Facebook as well… this emotive example is seriously flawed”

And these connection records are only going to get less relevant over time - an increasing number of websites including Facebook and Google encrypt their website under "https", which would make finding the name of the website visited far more difficult.

2. …but they’re still a massive invasion of privacy

Even though these records may be useless when someone needs to be found or monitored, the retention of Internet Connection Records (ICRs) is still very invasive – and can actually yield more information than call records, which Theresa May has repeatedly claimed are their non-digital equivalent.

Kennard notes: “[These records] can be used to profile [individuals] and identify preferences, political views, sexual orientation, spending habits and much more. It is useful to criminals as it would easily confirm the bank used, and the time people leave the house, and so on”. 

This information might not help find a missing girl, but could build a profile of her which could be used by criminals, or for over-invasive state surveillance. 

3. "Internet Connection Records" aren’t actually a thing

The concept of a list of domain names visited by a user referred to in the bill is actually a new term, derived from the “Call Data Records" collected by hone companies. Compiling them is possible, but won't be an easy or automatic process.

Again, this strongly implies that those writing the bill are using their knowledge of telecommunications surveillance, not internet era-appropriate information. Kennard calls for the term to be removed form the bill. or at least its “vague and nondescript nature” made clear.

4. The surveillance won’t be consistent and could be easy to dodge

In its meeting with the ISPA, the Home Office implied that smaller Internet service providers won't be forced to collect these ICR records, as it's a costly process. But this means those seeking to avoid surveillance could simply move over to a smaller provider. Bit of a loophole there. 

5. Conservative spin is dictating the way we view the bill 

May and the Home Office are keen for us to see the surveillance in the bill as passive: internet service providers must simply log the domains we visit, which will be looked at in the event that we are the subject of an investigation. But as Kennard notes, “I am quite sure the same argument would not work if, for example, the law required a camera in every room in your house”. This is a vast new power the government is asking for – we shouldn’t allow politicians to play it down.

6. The bill would allow our devices to be bugged

Or, in the jargon, used in the draft bill, subjected to “equipment interference”. This could include surveillance of all use of a phone or laptop, or even the ability to turn on its camera or webcam to watch someone. The bill actually calls for “bulk equipment interference” – when surely, as Kennard notes, “this power…should only be targeted at the most serious of criminal suspects" at most.

7. The ability to bug devices would make them less secure

Devices can only be subject to “equipment interference”, or bugging, if they have existing vulnerabilities, which could also be exploited by criminals and hackers. If security services know about these vulnerabilities, they should tell the manufacturer about them. As Kennard writes, allowing equipment interference "encourages the intelligence services to keep vulnerabilities secret” so they don't lose their own access to our devices. Meanwhile, though, they're laying the population open to hacks from cyber criminals. 


So there you have it  – a compelling soup of misused and made up terms, and ethically concerning new powers. 

Barbara Speed is a technology and digital culture writer at the New Statesman and a staff writer at CityMetric.