The Bot Wars: why you can never buy concert tickets online

Enterprising programmers are creating bots that can reserve, and in some cases buy, everything from restaurant tables to eBay goods before humans can even get a look in. Where will the bot wars end?

Diogo Monica was frustrated. All he wanted was to book a table at his favourite restaurant, but it seemed to be an impossible task.

State Bird Provisions, in the Fillmore district of San Francisco, was fully booked for 60 days in advance, and had been ever since last August, when Bon Appétit magazine named it the best new restaurant in the country.

No matter how many times he returned to the website, the answer was always the same: “No reservations are currently available. As tables become available, they will be shown here.”

Sure, the crispy spiced quail is “unparalleled”; yes, the rose-geranium ice cream is “a singular delight”; but could the promise of such culinary gems really rouse so many to try, every single day, that a table became literally unbookable?

Diogo wasn’t the only one having problems, a myriad of online complaints attested to that. “It’s the hardest place to get a seat in the city right now, full stop," said one. “Don’t even bother,” said another. “Tried every day last week = 0 bookings.” 

Where could all the reservations be going? Were they even going up at all?

Unlike most customers, however, Diogo had another weapon in his arsenal. As a computer programmer, currently working for Twitter founder Jack Dorsey's new venture Square, he had the skills to build a bot.

“A bot is essentially a computer program that automates the process that a human would go through when making a reservations,” Diogo explained via email. This bot would scan State Bird Provisions’ reservation page at regular intervals, emailing him every time it was updated. “I set it to run once every minute.”

At first he had some success. The resulting flurry of emails showed him that reservations really were being released, albeit at four in the morning. Other people were on to this too: each day, without exception, every table would be snapped up within the hour. Armed with this knowledge – and the occasional tip off about a last-minute cancellation – Diogo began to get reservations again.

It wasn’t fair – that he knew. Other fans without his technical skills were still going without a table. But it was a relief to have some success at last after expending so much time and energy on the project. 

But soon, even armed with this insider knowledge, the tables were being grabbed from under him – even the unexpected, irregular cancellations. “The moment they became available they would be gone in the time it took for the website to load.”

He realised that his bot was up against something more powerful. Someone else – or maybe several others – had created a bot that could both watch for and then reserve the available tables all by themselves.

“You fight fire with fire,” he said. “So I decided to fight back, creating my own.”

And so he did. The days when Diogo couldn’t get a table for love nor money are now a distant memory. His bot is so good at its job that sometimes he gets two or three reservations in his name in the same day – he thinks he’s now been to the restaurant “around 15 times”. But he doesn’t expect to stay top for long. If he wants to keep getting tables at State Bird Provision, he’ll have to constantly modify his bot as rivals are adapted to undercut him.

Diogo wrote about the "bot wars" on his website – releasing the code for anyone who wanted, in a bid to “level the playing field” for the less technically-minded.

Soon his competitors were coming out of the woodwork too, including Anil Bridgpal, a former Comcast software engineer.

News spread quickly among the foodie community: his tale of automatons, of reservations made in superhuman speed, rang true with many who had spent fruitless hours refreshing and refreshing the reservation page.

“Weird things go on,” agreed one thwarted diner, who had met with precisely the same problem with a restaurant 60 miles away. “I submitted my reservation within two-tenths of a second after midnight and was told I was too early. Within three seconds I resubmitted and the message said nothing was available. This is not the first time this has happened to me.”

It wasn’t just him, Diogo realised; it wasn’t just this restaurant.

***

Diogo’s dispatch from the frontlines of the new bot wars provided a rare insight into an underground community of unknown, unseen programmers who compete to build the fastest, most powerful bots in a virtual arms race.

Just as high frequency trading, via automated software, took over the financial markets in the early 2000s, the use of bots is a technique that is increasingly coming to dominate online sales of all stripes.

Diogo admits to deploying bots himself not only for dinner reservations, but to secure cinema tickets on busy opening nights and the cheapest flights to visit his family in Portugal. Anil knows of bots for “concert tickets, camping grounds and marathon registrations”. Such programs “are pretty easy to for any developer to write,” he told me.

Yet despite their ubiquity, in hacker circles there is little discussion of the ethics of bot use. There are no gentleman’s agreements to abstain from silicon wizardry when fighting it out with the general public. When there’s an online race, it’s every man for himself – and no one said the race would be fair.

While this might still be only a mild irritation when booking restaurants – where bot use is confined to only the most popular places – it already poses an enormous problem for the music industry. Online ticket sales for concerts are increasingly dogged by accusations of foul play from fans who feel they have been leapfrogged by so-called “scalpers” who buy up dozens of tickets as soon as sales open, then sell them immediately on, for a profit.

When Beyoncé announced an eleven-date UK leg of her 2013 “Mrs Carter” tour, fans knew that competition for the tickets would be fierce. So, on the morning of the 23 February, before the sun had risen, TV researcher Gemma Meredith was ready at her computer, the booking page on her laptop screen and her credit card on the table in front of her.

By 9.29am, she was refreshing the page repeatedly, jostling for first place in the queue. At 9.30 exactly she had been redirected to a "virtual waiting room" – then minutes later, the verdict: “Tickets could not be found. Please try again.”

So she did. Again, then again. Twelve minutes later, while she was still waiting, Ticketmaster announced that every single ticket had been sold. “It was incredibly frustrating,” she said. “Who was buying these tickets faster than I could reload the page? I don’t see what else I could have done.”

Across London, events manager Philippa Brady was in exactly the same position. “Me and a friend were both trying on two different computers – and neither of us got anything. I'm usually pretty good at getting tickets, but the Beyoncé sale was a joke.”

Even Olympic swimmer Rebecca Adlington had no luck, telling her followers on Twitter: “Three computers, refreshing like crazy but no Beyonce tickets! Hope anyone else trying has had more success!”

Yet within moments of the tickets selling out, touts were already offering them on eBay for as much as £1,000 for a standing ticket. “"There were so many, someone was obviously making a lot of money. But what I didn't understand was how they had bought so many when I couldn't even get one," said Gemma.

Ticketmaster, like all the other major ticket outlets, is in the midst of a huge battle with tout-controlled bots. During the Beyoncé sale alone, it thwarted 120,000 ticket requests from bots, and the number that got round the defences cannot be known.

A spokesman for the company said: “Attempts are made during every busy on sale to purchase tickets unfairly using bots. This is an arms race between the technology companies like us and the individuals who create bots. These people are continually innovating and refining bots to make them harder to identify. 

“Ticketmaster is committed to investing in technology that thwarts these bots and we will continue to work hard to ensure that tickets go directly into the hands of genuine fans.”

In the US, bots are thought to account for 90 per cent of traffic to the Ticketmaster website, and 60 per cent of ticket sales to some of the most desirable events. Equivalent statistics were not available for the UK, the spokesman said.

Ticket scalpers can be found advertising online, seeking programmers to develop ever more advanced software. One advert, now ended but still visible online, requests a program to “automatically search for tickets to events on Ticketmaster.com the moment they go on sale”. It must be able to “solve CAPTCHA screens with 75 to 100 per cent accuracy” and evade “cookie or IP conflicts that will lead to Ticketmaster blocking me”. A freelancer based in India was paid $450 for the job.

Indeed, anyone with the money can now buy ready-made bots from hackers "off the shelf". Websites like ticketbots.net offer, for a price, ticket-buying bots to suit a number of different sites - including that of Ticketmaster (£645), eBay-owned Stubhub (£385) and even the Royal Albert Hall (£490).

“Grabs tickets and hold for you instantly,” the Royal Albert Hall bot promises. “Specify section and/or row. . . automatically purchases tickets as soon as they are found.” It is the modern day equivalent of paying someone else to queue. The botmaker declined to reveal how many he had sold.

Yet despite this burgeoning market, some benevolent developers have been willing to use their skills for the greater good.  Adam Naisbitt, an entrepreneur from Milton Keynes, was so frustrated by the difficulties of buying tickets to the London Olympics that he created a bot to notify him when seats became available. But instead of keeping it to himself, he released the information on Twitter, under the @2012TicketAlert username.

“The response was overwhelming,” he said. “We had over 100,000 followers, reached over 2.5m people, helped thousands get tickets and raised over £2,000 for the Olympic Foundation.”

The popularity of his feed, and the success of bots more generally, comes down to the fact that these (often quite simple) automated programs can surpass human abilities in several ways. They are faster, making instantaneous ticket choices, or lodging eBay bids in the final milliseconds of a sale, and capable of making more rational decisions that humans – which has led to them being marketed for use in online gambling. “They do not make mistakes or stake more money because they believe a particular team or horse will win,” said Rade Jaramaz, director of a British-based company that develops bots for use on the Betfair exchange.

Some bots are simply successful because they don’t get bored, and will repeat the same action, again and again, without a drop in accuracy or motivation.

For this reason, many hackers built their first bots for use in computer games. Online games often feature a requirement to complete repetitive tasks to earn virtual wealth, such as mining for gold – the profits of which can then be traded for in-game bonuses or even real-life money. By writing a program to automate this process, a player can benefit from hours of mining without having to do any of the boring work themselves.

Just as in real life, the effect of the bots was to skew the market. For those who use the bots, your task becomes easy – whether that be mining virtual gold or buying real, sought-after concert tickets. For those that don’t, well, they are at an enormous disadvantage.

In many of those games, the same pattern emerged. As more and more bots came online, the situation spiralled out of control. Inflation skyrocketed as the gold-rich bot-owners priced out everyone else. Some games, like Diablo III, saw a total collapse of their virtual economy.

In the end, for the rule-abiding players, it stopped being fun. It stopped being fair. So they stopped playing altogether.

Such is the challenge facing the online sales industry. Every time a wall is built to keep out the bots, the hackers will be programming around it, popping up behind it, whack-a-mole style. But unchecked, the bots will cause havoc, undercutting the real fans, who will be forced to buy from touts, at spiralling prices.

Too many disappointments will see customers turning away altogether. Or, perhaps, they will buy their own software, hoping to beat the scalpers at their own game.

And thus the bot wars shall escalate. 

Is it fair that programmers can build bots to buy things you want? Photo: CJ Isherwood on Flickr, via Creative Commons
Cal Flyn is a freelance journalist, who writes for the Sunday Times, New Statesman and others. Find more of her work at www.calflyn.com and her Twitter handle is @calflyn.
Getty
Show Hide image

Chuka Umunna calls for "solidarity" among Labour MPs, whoever is voted leader

The full text of shadow business secretary Chuka Umunna's speech to Policy Network on election-winning ideas for Labour's future, and the weaknesses of the New Labour project.

There has never been an easy time to be a social democrat (or “democratic socialist” as we sometimes call ourselves in Britain). Whereas the right can demonise the poor and extol the virtues of the market, and the hard left can demonise the market and extol the role of the state, our position of constraining the domination of markets and reforming the state is, by definition, more complex.

It is nonetheless the case that social democracy has a historic responsibility, in every generation, to renew democracy and preserve a civic culture. This is achieved not through soundbites and slogans, but through the hard-headed development of a progressive politics that reconciles liberty and democracy, new comers and locals to our communities, business and workers, in a common life that preserves security, prosperity and peace.  This historic mission is all the more urgent now and my determination that we succeed has grown not weakened since our election defeat last May.

But, in order to be heard, it is necessary to make balanced and reasonable argument that both animates and inspires our movement, and which is popular and plausible with the people.  The first is pre-requisite to the second; and there is no choice to be made between your party’s fundamental principles and electability. They are mutually dependent - you cannot do one without the other.

We are in the midst of choosing a new leader and it is clear to anyone who has watched the UK Labour Party leadership election this summer that amongst a significant number there is a profound rage against Third Way politics – as pursued by the likes of Bill Clinton, Tony Blair, Gerhard Schröder and others - as a rejection of our fundamental values.

In the UK there is a view that New Labour accepted an uncritical accommodation with global capital that widened inequality, weakened organised labour and we were too close to the US Republicans and too far from the European left.

I do not believe this is fair, not least because we rescued many of our public services from the scrap heap when we came to office in 1997 and there were very significant achievements  we should celebrate.  New Labour renewed our National Health Service in a fundamental way; we built new schools and improved existing ones; we set up new children’s centres all over the country; we brought in a National Minimum Wage; we worked with others to bring peace to Northern Ireland; we introduced civil partnerships.  Just some of our achievements.

However, though we may take issue with the critique, I do not think we can simply dismiss out of hand those who hold critical views of New Labour. Like any government, the New Labour administration made mistakes - it could and should have achieved more, and done more to challenge the Right’s assumptions about the world. In the end, it is not unreasonable to be ambitious for what your party in government can achieve in building greater equality, liberty, democracy and sustainability. It is far better we acknowledge, not reject, this ambition for a better world, as we seek to forge a new politics of the common good fit for the future.

Realising our values in office has been disrupted by globalisation and the surge of technological forces that are displacing and reshaping industry after industry.

Some argue that globalisation as an ideological construct of the right. But we must recognise that we live in an increasingly integrated world in which markets have led to an unprecedented participation of excluded people in prosperity, a rise in living standards for hundreds of millions  of people and a literacy unprecedented in human history – this is particularly so in emerging economies like my father’s native Nigeria. And the internet has led to a level of accountability that has disturbed elites.

Yet, this has been combined with a concentration of ownership that needs to be challenged, of a subordination of politics that requires creative rather than reactive thinking, and these global forces have exacerbated inequalities as well as helped reduce poverty.

So it is important that we understand the sheer scale and impact of new technologies. At the moment we are engaged in a debate about Uber and its threat to one of the last vestiges of vocational labour markets left in London, those of the black taxi cabs and their attainment of 'The Knowledge'. But the reality is that within the next decade there will be the emergence of driverless cars so we have to intensify our exploration of how to support people in a knowledge economy and the realities of lifelong learning, as well as lifelong teaching. As people live longer we will have to think about how to engage them constructively in work and teaching in new ways.

Once again, I'm addressing all of this, Social Democracy requires a balanced view that domesticates the destructive energy of capital while recognising its creative energy, that recognises the need for new skills rather than simply the protection of old ones. A Social Democracy that recognises that internationalism requires co-operation between states and not a zero sum game that protectionism would encourage.

Above all, Social Democratic politics must recognise the importance of place, of the resources to be found in the local through which the pressures of globalisation can be mediated and shaped. Our job is to shape the future and neither to accept it as a passive fate nor to indulge the fantasy that we can dominate it but to work with the grain of change in order to renew our tradition, recognising the creativity of the workforce, the benefits of democracy and the importance of building a common life.  Sources of value are to be found in local traditions and institutions.

This also requires a recognition that though demonstration and protest are important,; but relationships and conversations are a far more effective way of building a movement for political change.

One of the huge weaknesses of New Labour was in its reliance on mobilisation from the centre rather than organising. It therefore allowed itself to be characterised as an elite project with wide popular support but it did not build a base for its support within the party across the country, and it did not develop leaders from the communities it represented. It was strong on policy but weak on strengthening democratic politics, particularly Labour politics.

Over half a million people are now members, supporters or affiliated supporters of our party, with hundreds of thousands joining in the last few weeks. Some have joined in order to thwart the pursuit of Labour values but many more have joined to further the pursuit of those values, including lots of young people. At a time when so many are walking away from centre left parties across the Western world and many young people do not vote let alone join a party, this is surely something to celebrate.

So it is vital that we now embrace our new joiners and harness the energy they can bring to renewing Labour’s connection with the people. First, we must help as many them as possible to become doorstep activists for our politics. Second, I have long argued UK Labour should campaign and organise not only to win elections but to affect tangible change through local community campaigns. We brought Arnie Graf, the Chicago community organiser who mentored President Obama in his early years, over from the U.S. to help teach us how to community organise more effectively. We should bring Arnie back over to finish the job and help empower our new joiners to be the change they want to see in every community – we need to build on the links they have with local groups and organisations.

I mentioned at the beginning that in every generation Social Democracy is besieged from left and right but the achievements of each generation are defined by the strength of a complex political tradition that strengthens solidarity through protecting democracy and liberty, a role for the state and the market and seeks to shape the future through an inclusive politics. Solidarity is key which is why we must accept the result of our contest when it comes and support our new leader in developing an agenda that can return Labour to office.

Yes, these are troubled times for social democrats. All over Europe there is a sense among our traditional voters that we are remote and do not share their concerns or represent their interests or values.  There is surge of support for populist right wing parties from Denmark to France, of more left wing parties in Greece and Spain and in Britain too. There is renewal of imperial politics in Russia, the murderous and abhorrent regime of ISIL in the Middle East, volatility in the Chinese economy and in Europe a flow of immigration that causes fear and anxiety.

But, the task of Social Democracy in our time is to fashion a politics of hope that can bring together divided populations around justice, peace and prosperity so that we can govern ourselves democratically. We have seen worse than this and weathered the storm. I am looking forward, with great optimism to be being part of a generation that renews our relevance and popularity in the years to come.

Chuka Umunna is the shadow business secretary and the Labour MP for Streatham.