The 20th century was the Conservative century, with the Conservatives remaining the natural party of government for most of it.
Before 1997, there were only three general elections that gave the left untrammelled power to do things that the Tories did not like - 1906 (though the Liberals faced a hereditary upper house with theoretically unlimited powers), 1945 and 1966. Yet, the Conservatives' defeat in 1997 was their worst since the time of the Great Reform Act, and they have now been in opposition for a longer continuous period than at any time since the 18th century. Why had a party with so strong an appetite for power seemingly lost the will to win?
David Frum, a former speechwriter to George W Bush, once wrote that "when a political party offers the voters ham and eggs and the voters say, 'No, thanks,' its first instinct is to say, 'OK then. How about double ham and double eggs?'" Three successive Conservative leaders - William Hague, Iain Duncan Smith and Michael Howard - responded to defeat by seeking to mobilise the Tory "core" vote. They did so by highlighting the "dog-whistle" issues: the so-called "Tebbit trinity" of Europe, immigration and taxes, or what Tim Bale calls "the politics of the 19th tee".
The trouble was that the Conservative core vote was too narrow, being based on the over-65s and the geographically and socially immobile, to act as a springboard for electoral revival. In 2001, there were actually swings away from the Tories among the professional and managerial classes, 25- to 34-year-olds and ethnic minorities, groups from which they already had low levels of support in 1997. The party seemed defined largely by what and whom it hated - the euro, the European Union, asylum-seekers, gay people and criminals. No wonder that the Conservatives were seen, in the words of the then party chair, Theresa May, as the "nasty party".
The Conservatives did better in 2005, coming just 3 percentage points behind Labour. In England, they actually won more votes than Labour but, thanks to an electoral system that they alone among the major parties defend, secured 93 fewer seats. Some argued, as Winston Churchill had done in 1950, that "one more heave" would suffice.
But David Cameron appreciated that the situation in 2005 was quite different. The Conservatives had indeed made gains in the south of England, but they had lost support in the key marginals in the West Midlands and the north-west. They had done particularly badly among the professional classes, which had usually voted for them - support for them among this group had fallen from 54 per cent in 1992 to just 18 per cent, while their support among women had fallen from 45 per cent in 1992 to 32 per cent. Among university graduates, the Conservatives were now the third party, behind the Liberal Democrats.
Of the 31 seats that the Conservatives did win back from Labour, the majority had gone Conservative as a result of Labour voters switching to the Liberal Democrats and other parties, not to the Tories. Survey evidence showed that 79 per cent of Conservative voters thought their party was "on the right track to get back into power before long", but that just 28 per cent of those who had not voted Conservative agreed. The Tories remained, it seemed, unelectable.
Yet all that has now changed. For the first time since Margaret Thatcher's heyday in the 1980s, the Conservatives have enjoyed a continuous lead in the opinion polls. They are once again serious contenders for power. So how did the party manage to recover so rapidly?
Tim Bale, in this history of the Conservative Party from the defenestration of Thatcher in 1990 to the present, seeks to analyse both the collapse and the recovery. His book is painstaking and reliable, if somewhat humdrum, and is marked by a failure to distinguish between the important and the trivial. Who now cares, for example, that in spring 1999 Peter Lilley, as Tory shadow chancellor, made a speech urging a return to Thatcherite values, and that it caused ructions among those close to the then leader, William Hague?
Nevertheless, Bale makes clear the reasons for the current leader's success. "Cameron," he writes, "not only could read the polling evidence, [but] he was the first Tory leader in well over a decade who was realistic enough to act on it." In essence, Cameron has decontaminated the party. He has emphasised issues not historically associated with the Conservatives, such as the environment, childcare and work/ life balance - "touchy-feely waffle", in the words of the Daily Mail. Unlike his predecessors, he has consistently supported the public services, stressing his family's use of the National Health Service and state schools. He understands that voters do not want a strong economy at the expense of such systems.
Conservative leaders, therefore, no longer question the safety net of social welfare that alone makes the pressures of globalisation tolerable. For globalisation inevitably increases uncertainty and risk, with new jobs being created as old ones disappear. Voters will accept such uncertainties only if they are combined with strong welfare policies, so that they do not lose everything when markets move.
Helped by the credit crunch, Cameron has made the economy once again a Conservative issue. In 2005, Patricia Hewitt taunted the Tories, saying that where they once "used to run on the economy . . . Mr Howard's Tory party just runs away from it". That, too, has changed. The Conservatives now poll marginally ahead of Labour as the party best trusted to run the economy, even though the recession has exposed the bankruptcy not of state regulation, but of the rule of markets.
Cameron's great strength is that he has never been an ideological politician. Twenty-five years ago, he began his university career at Brasenose College, Oxford, where I was his tutor. An outstanding undergraduate, he avoided politics, concentrating instead on his studies and on getting to know his fellow students, among whom he was extraordinarily popular. He has not, in essence, changed. He is a Conservative in the Macmillan, not the Thatcher mould - pragmatic and socially concerned, but not committed to any fixed blueprints.
The obverse of this strength is that it is sometimes difficult, as it was with Macmillan, to discern any clear sense of direction. Cameron has not yet provided his party with the narrative that it so badly needs. In addition, his attempts to persuade constituency associations to choose more female and ethnic-minority candidates have not always endeared him to Tory activists. One Conservative has compared the party to "a British telephone box, which looks appealing on the outside, but if you open the door, it smells really bad". That is why, as Cameron himself concedes, the Conservatives have yet to "seal the deal" with the British people, and why the outcome of the next election remains open. But he is, nevertheless, the most formidable threat that the left has faced since the advent of Thatcher.
Vernon Bogdanor is professor of government at Oxford University. He is the editor of “From the New Jerusalem to New Labour: British Prime Ministers from Attlee to Blair" (Palgrave Macmillan, £20)