A London Underground train enters Oxford Street station, below a realtime indicator giving passengers information about delays or cancellations. Photo: Getty Images
Show Hide image

Public transport bodies: producing lots of data, not necessarily making the most of it

The trend over the last few years has been for public transport authorities to accept that their data should be made public - while at the same time letting the private sector absorb the cost of making use of it.

There’s an app called Citymapper that is probably the one thing I would consider necessary to my day-to-day life, and the justification for owning a smartphone. The things I do on the move - listening to music, checking the web, etc. - are things I don’t need to do while moving; but knowing where I am and where the thing I am moving towards are in relation to each other is an everyday essential.

Obviously, this is what maps are for, and there are maps all over London, but it’s a big city, and coming to it for the first time can be very confusing. Transport for London has a journey planner on its website, but it often feels very unintuitive. For example, here’s what happens when you ask for help with a journey that a daytripper to the capital might take - from Paddington to Tottenham’s football ground, White Hart Lane:

Before I'd moved to London and learned the language of the tube or bus maps, and was merely a tourist, I found this format incredibly confusing. I was using the journey planner because I didn’t know how to get around the city, yet I was being given a list of places I’d never heard of and told to travel between them without any indication of where they were relative to each other. For journeys with lots of steps, or which involve buses, this format can make simple journeys seem convoluted. (And while the TfL site has had a redesign recently to clean it up, it's still the same basic idea.)

Citymapper, though, does this:

It's the same information presented as a map instead of a step-by-step list. It offers text instructions as well, but the key thing is it shows the user where they are and where they’re going, and where the stops along the way are. (And, when I went to New York City earlier this year on holiday, it was better there than the official step-by-step journey planner too.)

Citymapper doesn’t have a special data source - it’s simply using the official data that these transport systems generate in a different way. TfL, like many transport authorities, has what's known as an application programming interface - or API - which tells external apps and sites how to communicate with its servers, and how to retrieve the information they need. This isn't just trains, but also bus departure times, bike hire docking station statuses, taxi information, fare breakdowns, and so on, and is why a search on Apple’s App Store or Google’s Play Store for "London transport" delivers dozens of results.

It's left me wondering this: why is it that public bodies like TfL are so good at producing data, but so bad at using it? Why is an organisation like TfL - which has such a strong brand identity, and which puts so much care and attention into its tube maps, for example - happy to sit back and let third parties control how the public experiences its information?

I emailed Azmat Yusuf, Citymapper’s founder, about this, and he got back to me:

Wouldn't say transport authorities are bad at using data, they are focused on infrastructure running on time which requires utilising their own data. We are focused on apps and we like to do that well.

Most transport authorities are opening up data, some more reluctantly than others, since it’s a cost to them. But starting to believe and understand that they are better off letting others like us develop applications. Also there's an argument that the data belongs in the public domain anyway."

It’s an interesting idea, that the data “belongs” in the public domain. TfL is a government body, and in a democracy that should mean that we - the public - have a right to its data. Yet we’re living through a strange time, when privatisation and outsourcing make it hard to define what counts as government and what as private company; and where the government is committed to starving TfL of subsidies so that it has to fund itself entirely out of fare revenues, so that it operates less like a public utility and more like a private firm.

Perhaps we should simply be grateful that TfL goes to the lengths it does to provide such comprehensive real-time data feeds across its jurisdiction. The situation could be worse - as I discovered when I spoke to Tom Cairns, founder of Realtime Trains.

His site uses a large range of data sources to piece together a picture of exactly how a train moves around the UK relative to its timetabled route. The problem is that those data feeds are highly reliant on human input to reflect things like cancellations and delays - and if somebody doesn't keep updating those feeds as fast as new information comes in, the official sources, and by extension Realtime Trains, would return innacurate information.

Cairns and his two colleagues use the knowledge of how the network operates - from the lengths of trains to their top speed, to how fast signals change on certain track sections to how long it takes a train to pull away from the platform - to model the likely future arrival time of a train relative to how well it’s already done in its journey. “I have people who go around the UK noting down the timing of the wheels after they start and compare that to the signalling equipment, to get a more accurate picture of what is happening,” he told me. “There’s a lot of work that’s gone into improving the accuracy of the information by hand.”

The problems he faces, though, illustrate the difficulties small developers face when trying to work with a large institution with a reluctance to share its secrets.

When I get to train companies about getting information directly they say ‘why should we give it out to you, a tiny third party on the periphery?’ In their industry, it’s slow moving. They never had competition in this field either, and they didn’t see a problem with what they’re doing. The opening of [data] took many of us by surprise, in part because we knew they didn’t see a problem.”

Cairns also says that good ideas on third party sites, like his map of track obstructions, have a habit of popping up on the official journey planners a few months down the line. "It was added to the National Rail development roadmap," he explained. "But of course it’s three years later and it still hasn’t appeared." Institutional willingness countered by institutional lethargy.

TfL is due to update its API in the autumn to make it adhere to more universal data standards, and theoretically make it easier for developers to use. I emailed TfL to ask what the motivation was for letting third-party developers loose on its data - whether, for example, it was because of budget limitations - but digital PR lead Rubin Govinden told me that they don't comment on what other people have done.

He did say this, though: 

There could be eight different travel apps out there and they could all be offering their own service, but the one thing they have in common is they're using the most accurate and up-to-date information. They don't have to wonder ‘is this data pukka?’ - yes it is, yes it's up-to-date, they're all going to be saying the exact same thing at the same time in the same way. What we're giving developers is the ingredients they need to produce the products that our digital customers want.”

When pushed for more specifics, he repeated that he didn’t want to name any individual app or site: “If it's to work successfully, lots of people have to work together, and we're doing our bit by making our data freely and openly available in as many formats in a recognisable format that can be used by everybody.”

This echoes Yusuf's views - TfL's focused on the bit of its job that involves running the trains on time, and the data bit is the afterthought. It's perhaps one model that other public bodies - and not just train companies - could look to when figuring out just how much they want to commit to developing their own tools to make use of the data they produce.

Ian Steadman is a staff science and technology writer at the New Statesman. He is on Twitter as @iansteadman.

Paul McMillan
Show Hide image

"We're an easy target": how a Tory manifesto pledge will tear families apart

Under current rules, bringing your foreign spouse to the UK is a luxury reserved for those earning £18,600 a year or more. The Tories want to make it even more exclusive. 

Carolyn Matthew met her partner, George, in South Africa sixteen years ago. She settled down with him, had kids, and lived like a normal family until last year, when they made the fateful decision to move to her hometown in Scotland. Matthew, 55, had elderly parents, and after 30 years away from home she wanted to be close to them. 

But Carolyn nor George - despite consulting a South African immigration lawyer – did not anticipate one huge stumbling block. That is the rule, introduced in 2012, that a British citizen must earn £18,600 a year before a foreign spouse may join them in the UK. 

“It is very dispiriting,” Carolyn said to me on the telephone from Bo’ness, a small town on the Firth of Forth, near Falkirk. “In two weeks, George has got to go back to South Africa.” Carolyn, who worked in corporate complaints, has struggled to find the same kind of work in her hometown. Jobs at the biggest local employer tend to be minimum wage. George, on the other hand, is an engineer – yet cannot work because of his holiday visa. 

To its critics, the minimum income threshold seems nonsensical. It splits up families – including children from parents – and discriminates against those likely to earn lower wages, such as women, ethnic minorities and anyone living outside London and the South East. The Migration Observatory has calculated that roughly half Britain’s working population would not meet the requirement. 

Yet the Conservative party not only wishes to maintain the policy, but hike the threshold. The manifesto stated:  “We will increase the earnings thresholds for people wishing to sponsor migrants for family visas.” 

Initially, the threshold was justified as a means of preventing foreign spouses from relying on the state. But tellingly, the Tory manifesto pledge comes under the heading of “Controlling Immigration”. 

Carolyn points out that because George cannot work while he is visiting her, she must support the two of them for months at a time without turning to state aid. “I don’t claim benefits,” she told me. “That is the last thing I want to do.” If both of them could work “life would be easy”. She believes that if the minimum income threshold is raised any further "it is going to make it a nightmare for everyone".

Stuart McDonald, the SNP MP for Cumbernauld, Kilsyth and Kirkintilloch East, co-sponsored a Westminster Hall debate on the subject earlier this year. While the Tory manifesto pledge is vague, McDonald warns that one option is the highest income threshold suggested in 2012 - £25,700, or more than the median yearly wage in the East Midlands. 

He described the current scheme as “just about the most draconian family visa rules in the world”, and believes a hike could affect more than half of British citizens. 

"Theresa May is forcing people to choose between their families and their homes in the UK - a choice which most people will think utterly unfair and unacceptable,” he said.  

For those a pay rise away from the current threshold, a hike will be demoralising. For Paul McMillan, 25, it is a sign that it’s time to emigrate.

McMillan, a graduate, met his American girlfriend Megan while travelling in 2012 (the couple are pictured above). He could find a job that will allow him to meet the minimum income threshold – if he were not now studying for a medical degree.  Like Matthew, McMillan’s partner has no intention of claiming benefits – in fact, he expects her visa would specifically ban her from doing so. 

Fed up with the hostile attitude to immigrants, and confident of his options elsewhere, McMillan is already planning a career abroad. “I am going to take off in four years,” he told me. 

As for why the Tories want to raise the minimum income threshold, he thinks it’s obvious – to force down immigration numbers. “None of this is about the amount of money we need to earn,” he said. “We’re an easy target for the government.”

Julia Rampen is the digital news editor of the New Statesman (previously editor of The Staggers, The New Statesman's online rolling politics blog). She has also been deputy editor at Mirror Money Online and has worked as a financial journalist for several trade magazines. 

0800 7318496