Records of Broughton in Buckinghamshire date back to the Domesday Book, the first medieval census - a comprehensive account of hitherto uncollected personal information. Nearly a millennium later, the online search giant Google was in the former village (now a suburb of Milton Keynes) updating records for its online mapping tool, Street View. A little after 9am on a Wednesday morning, an unmarked black Vauxhall Astra was spotted with a camera on a metre-high pole. People didn't like that "it could see over their garden walls", recalls the local councillor John Bint.
“The car was parked here on the corner," explains Edward Butler-Ellis, a member of the local Neighbourhood Watch. "A crowd gathered around it, but [it was] not the barricade with pitchforks and so forth that [the media] were so intent on promoting. It was a perfectly civil discussion." By the time the police arrived, the car was gone. "The story escalated way beyond what we'd anticipated. We just wanted the chap to go away."
By Friday morning, the event had generated thousands of hits on Google News. "A lot of people didn't realise what was happening," Butler-Ellis says. "My gran had never heard of Google really. When I told her about the cameras, it made her blood boil. 'How dare these people come and take photographs of my house without asking me?' she said."
The controversy seemed to surprise Google; the firm has long and loudly professed its objective to gather user and other data in a quest “to organise the world's information". The company is so good at this that David Cameron wants to put it in charge of organising our medical records. Its products don't just use data, they help us find it - and we continue to ask for help. When we use its software, and with our permission, Google's "bots" follow us everywhere we go: as we search the internet, look up an address, reply to an email, make a purchase, chat with our friends online or walk about with our mobile phone. Though it does not recognise you as an individual, Google's software could soon track and store almost everything you want - and everything you do.
The Google story begins in 1996 with Sergey Brin and Larry Page, two young engineers newly enrolled in a PhD programme at Stanford University in California. The web was in its infancy; finding information online was as cumbersome as wading through the Encyclopaedia Britannica, and often far less decent. Brin and Page had an idea. They chose to look at links between websites instead of just site content, as other search services did. If you, say, searched for "coffee", the old search engines would give you the pages with the most mentions of "coffee". Google gave you the pages that had the most links to them from other sites. So you might get a page that mentioned "coffee" less often, but was deemed more authoritative by previous visitors.
Within a year, the pair had raised a million dollars to develop their project, but then faced the challenge of finding a way to make money out of it. Their solution - charge companies to advertise on specific searches, right next to the results.
The pair's innovation transformed Google into a new type of company that has defined the internet, one that organises information in order to make money from advertising, knowing what users want and telling them how to get it. That's why they needed the photos from Milton Keynes: for a service that, every day, shows millions of people how to get to where they want to go and what they want to find, along with-as part of the Google Maps service-a carefully targeted advert or two.
The model has put Google in conflict with privacy campaigners. Gus Hosein, a senior fellow at Privacy International, sees it less as a search or email company than an advertising company. "That's how they make all their money," he says. Certainly, Google's growth since 2000 has been about finding ways to advance beyond search ads, advertising against thousands of other web pages, including its own email service.
In each case, Google uses information we provide to serve us the right advertisements at the right time. Sometimes, as in Gmail (now Googlemail in the UK and Germany), it's the words we're reading that drive the advertisements: a friend writes that he's livid about a recent football loss, and you see an advert for football tickets. On our mobile devices, it's the GPS tracking that could enable Google to serve up location-based ads.
Hosein thinks Google is working hard to build privacy checks into its system; no human being ever reads your email or scans your searches. But the expansionist vision of Google's chief executive, Eric Schmidt, troubles him. There are 600 million internet-enabled mobile phones out there; that could give Google an even richer seam than search. Phones are very personal in the data they collect. As Schmidt told Maria Bartiromo in BusinessWeek last month: "If we know a fair amount about a person, with their permission, we can target a useful ad - you know, 'It's Eric. You had a hamburger yesterday, do you want pizza today? There's a pizza store on the right.' That kind of ad is likely worth a lot of money to an advertiser because it will generate a sale."
It could be worth a lot to Google, too, enabling it to expand its prowess beyond PCs. With nine billion searches on its pages a month, Google has a 65 per cent share of the US search market, according to the online data-tracker comScore. Its next largest competitor, the once-mighty Yahoo!, has just fallen below the 20 per cent mark. Here in the UK, Google's dominance is even more pronounced: another data-tracker, Hitwise, puts Google's share at 90 per cent, followed by a handful of competitors with less than 3 per cent each.
In other areas of advertising, such as banner ads on partner pages, no one player has a clear majority share yet. But data from Credit Suisse and Evercore Partners suggests that Google is poised to grasp one, showing the fastest growth in each category of advertising, as well as the largest single share (31 per cent) of all online advertising. In the UK, the situation is even more pronounced. As Nigel Gwilliam of the Institute of Practitioners in Advertising puts it, "You can get figures for Google's total revenue in the UK from their financial statements [and] compare their total income to the total UK market. Certainly, the last time I looked, they were in their own right half the [online advertising] market. Search is traded as a dedicated marketplace, and within that they were almost the market."
Google's unparalleled popularity makes its targeting technology even more valuable to advertisers, because having more users delivering collective wisdom about what they find relevant increases the chances of a good advertising match. How much more valuable is hard to quantify, but Google took $21bn in advertising revenue last year, and managed to make a profit of almost $8bn. That represents 30 per cent growth since 2007 and a 40 per cent operating margin - an astonishing performance in any year, but especially during a global recession.
Though some of the protesters in Broughton might have been concerned that Google was out to make a profit from the images of their homes, there is nothing wrong, in legal terms, with it doing so - especially as, in this case, the photos were taken from a public street. On Google's home turf in the US, there are no privacy laws that specifically govern online user data, just a host of voluntary compliance guidelines. For the most part, these guidelines permit all forms of data collection, so long as companies don't lie to consumers about what is being recorded. The European Commission goes a little further: it bars companies in EU member states from holding personal data longer than their express business purpose requires. Google's express business purpose, Hosein notes, is to hold the data we provide in the online "cloud", where it may stay indefinitely. As Schmidt's vision of hamburger-hawking mobile phones reveals, Google is not apologetic about its business model.
Hosein is critical of what he sees as the failings of a legal order built on what companies say they will do, rather than explicit restrictions on what they can do, which means that they can only be penalised for breaking their own commitments - that is, for holding more data, for a longer period of time or for a different use, than they initially promised. The penalty a company faces for going against its word is a flat £5,000 in the UK; in the US, it's just a verbal slap on the wrist.
Yet data aggregation and targeted advertising - the practices that make Google so profitable, and so alarming to some - risk landing the company in a different sort of trouble: the quagmire of competition law. By being strategically so good at pursuing its founders' data-driven vision of the social good, might Google cross a legal line with its potential dominance in gathering and organising information? The question agitates the otherwise amiable Oliver Rickman, Google's UK spokesman. A casual post on the social networking site Twitter, requesting information from friends for this story, received a quick rebuttal from the company.
Rickman is not wrong to be worried. After all, antitrust law (as competition law is known across the Atlantic) succeeded in breaking up the world's largest oil company, Standard Oil, in 1911, and came close to victory against Microsoft a decade ago. That is why Google has invested in a top-of-the-line legal defence team, led by Dana Wagner, a young lawyer who cut his teeth at the US department of justice. Fresh from the older, greyer offices of federal regulators, Google's new recruit is fiercely committed to the company's mission and promises. He dismisses most antitrust arguments as the cynicism of competitors, or those who haven't seen the digital light. "One of the reasons the internet has been so successful and such an engine for growth and innovation is that there hasn't been a lot of regulation," he says. "There is certainly a role for sensible antitrust law, but it should be kept to a minimum."
The law is broadly similar in the US and the EU. Structured to protect companies from one another as much as to protect consumers, competition provisions (or antitrust laws) bar dominant companies from abusing their position to destroy rivals or overcharge customers, and seek to prevent rising stars in an industry from using unfair practices - such as price manipulation, customer coercion, collusion with rivals or merger activity - to secure dominance. Mergers and acquisitions involving dominant players that reduce competition are particularly closely scrutinised, even if the company hasn't otherwise abused its position.
All these provisions, however, begin with market definition: what is the company really selling and to whom? And has it done anything specific and deliberate to block out - or buy out - the competition or to manipulate price to the customer? Google says it hasn't, insisting that its dominance in search and search-based advertising is a function of its superior products and technology. Indeed, the company has even set up a "Data Liberation Front", apparently to ensure consumers are not "locked in" to its products. They remain with Google by choice. Richard Liebeskind, a former justice department lawyer who now specialises in antitrust litigation, explains: "If you gain a monopoly by skill, foresight and industry, none of that violates the antitrust laws." Google being big may be scary, but it's not illegal.
Nonetheless, in 2007, when Google decided to buy another advertising firm - DoubleClick - rivals including Microsoft and Yahoo! were quick to condemn the deal, arguing that it gave Google an unfair monopoly. By a vote of four to one, the US Federal Trade Commission decided it did not. But Google was now being watched. When, in 2008, it tried to strike a partnership with Yahoo!, the US department of justice decided it would investigate. Google abandoned the deal. Wagner claims that many challengers are just envious of Google's search skills. After all, rivals - led by Microsoft - have pooled their funds to get their own team, a venture euphemistically named Initiative for a Competitive Online Marketplace (ICOMP).
Momentum to challenge Google is building from smaller players, too, with different search technology. One firm, TradeComet.com, filed suit in February this year, alleging that Google had "terminated the voluntary course of dealing it had with SourceTool [TradeComet's affiliate] by manipulating its auctions so that SourceTool faced vastly higher prices to acquire search traffic, prices so high it was uneconomical for SourceTool to win auctions that it had routinely won prior to Google's exclusionary strategy". Google denies the allegations and is fighting them vigorously.
Wagner is unperturbed. "As a successful company, we expect scrutiny; we're not bothered by it," he says. Yet Christine Varney, newly appointed by President Obama to lead the antitrust division at the US justice department, has an eye on the company. "Microsoft are so last century," Varney is reported to have said before her appointment. "Google has a monopoly in online advertising."
Her first task will be to rule on the legality of a deal Google has struck with US copyright holders to digitise millions of old library books and make them available and searchable online. The endeavour is costing Google hundreds of millions of pounds and has won praise for its scope, but the deal includes clauses that could make it harder for anyone else to do the same. While publishers may offer similar agreements to other companies, they may not offer them better terms than Google now enjoys. In addition, in a quirk of the terms, Google obtained exclusive access to so-called orphan works - out-of-print books with expired copyrights where there is no copyright holder who can offer the books to other companies. The case goes to the heart of Google's core business model: acquiring access to, searching and then monetising data that is already out there. Like the gardens of Milton Keynes, what's valuable about the library to Google is finding out what users search for within it.
Predictably, Microsoft was the first to complain. David Wood of ICOMP says: "If antitrust law fails in this case, that means we need new regulations." The case for new regulations - or at least new case law - was previously made by Pamela Jones Harbour, the one US federal trade commissioner who dissented from the DoubleClick judgment in December 2007. (The FTC, incidentally, couldn't find her opinion, and advised that we use Google to search for it.)
Google successfully argued that, because DoubleClick was a display ad firm specialising in flashy banners and Google's ads were mostly tied to keywords and came in text form, the two firms weren't really in the same business, and so Google would not gain a significant advantage in the market. Harbour's view had been that Google was in the business of "targeted online advertising", and any and all data that advertisers want would eventually be useful to it. She explained: "Search information, gathered by Google, combined with browsing information [information on what sites we visit], gathered by DoubleClick, will create a far richer source of data to enable highly targeted advertising."
In Harbour's view, expansion that brought Google new data could further its dominance in targeting adverts to what we say we want. Whether it is adding new books to search pages that users visit or new features to Gmail, Google is expanding its share in the only online market from which anyone has worked out how to make money: online user data.
Harbour was arguing that Google's expanding share of user data would give it an unassailable advantage in targeting online adverts. In doing so, she was making the link between privacy concerns about Google's control of personal information and competition concerns about Google's dominance in online advertising. As Harbour wrote in her dissent, "After all, why would Google pay billions of dollars for DoubleClick out of the hands of competitors, if Google does not intend to combine the two firms' valuable datasets?"
At the time, Google insisted advertisers did not see it that way. Since then, the ground has shifted. As Wagner says: "Before, text-based search ads were like direct-response ads, where display ads were based on branding, but now people realise that there's a branding value to search ads and, as the technology improves, they do converge." Increasingly, Google uses the same insights about our preferences to target all internet advertising.
This is leading some lawyers to look back at Harbour's proposed remedies. In the case of Google and DoubleClick, she wrote, "The commission could have asked the parties to make binding commitments regarding their handling of data, and to memorialise those representations in a consent agreement." She suggested that the FTC adopt this standard as a "future approach to data mergers".
As Google's legal team points out, other companies such as Microsoft now see advertising based on user preferences as the main revenue source online and can seek to obtain, and monetise, user data, too. That hardly helps regulators and consumers who may want less, not more, surveillance. However, solving the problem is difficult: on the fast-changing web, predictions about what Google and its peers will do next are often shots in the dark. We know a whole lot less about their plans than they do about us.
Maha Atal is a business journalist in New York. Damian Kahya is a journalist based in London