Francis Bacon once remarked that he voted for the right because it makes the best of a bad job. Like many of the painter's observations, it captures a whole way of looking at things - in this case, one that has vanished from politics. The idea that humans are an incorrigibly flawed species is taboo today, but it was the right's key tenet for much of the past 200 years. It was abandoned only in the 1980s, when the Tory party became the vehicle for an anachronistic experiment in reviving 19th-century individualism.
The keynote of Margaret Thatcher's outlook was not any insight into human imperfection, but a militant faith in progress. What Britain - and later, as her vision became more hubristic and fantastical, the entire world - needed was a permanent revolution, in which the spirit of capitalism was promoted throughout society. Armed with this bullish philosophy, Thatcher forced a brutal modernisation on the country. It was a traumatic process that cost her the leadership and nearly destroyed her party. No Conservative leader after her has been able to slow the party's decline into a rancorous rabble. If John Major was unable to nudge it back into the mainstream, William Hague and Iain Duncan Smith fared no better in their attempts to lead it from the right. With the ties of loyalty severed by a matricidal coup, the party had become ungovernable.
Can Michael Howard succeed where three previous Conservative leaders failed? His emergence as leader has been greeted as a sign that the Tories are regaining their sanity. Howard is a subtle and ruthless politician of vast experience. Unlike his Cambridge contemporary Kenneth Clarke, he shows signs of having reflected on that experience. Assuming that his coronation is unopposed and does not provoke too much anger in the constituencies, he may be able to unify the party and turn it at last into something closer to an effective opposition - but can he take it back to power? His hardline Thatcherite past may help to endear him to Tory activists, it is often said, but it can hardly enable him to reach out to the floating voters he must attract if he is to lead the party back from the wilderness.
It is sensible to discount the media euphoria surrounding Howard's seemingly effortless rise to power. Recent opinion polls suggest that the Tories face forbidding difficulties, and even now may not have hit the bottom. If Howard fails to deliver the goods in terms of party unity, he will face the wrath of the constituencies, which remain resentful at the way Duncan Smith was removed from power. Even if Howard were to retain the loyalty of his party, it may be at the cost of failing to re- coup the voters it lost to new Labour and is now losing to the Liberal Democrats.
It is entirely conceivable that, a couple of years on, the Tories will still be locked in introverted despair, facing a third landslide defeat at the polls.
There is another scenario, however. Howard may turn out to be the Tory leader who lays Thatcher's ghost. His cast-iron right-wing credentials may enable him to make a decisive break with the party's Thatcherite past - an achievement that has consistently eluded the party's avowed modernisers. If he can do this, he will have a big advantage over Tony Blair's new Labour, which is mired in the defunct orthodoxies of the Thatcher era.
It is commonly thought that the only way the Tories can re-enter the political mainstream is by undergoing a Blair-style transformation. This is a distinctly unpromising strategy for the Tories. It would amount to copying Blair copying Thatcher - hardly a winning gambit at a time when the voters are looking for something very different from the neoliberal gruel that both parties have served up for the past decade.
Blair's initial success in attracting Tory voters may have come from aping Thatcher, but history has moved on since then. If the imperative for the Tories is to bury their Thatcherite past, re-importing Thatcherism from new Labour is hardly a wise move.
The vision of a "kinder, gentler" Conservative Party so dear to Tory modernisers is a view of politics from the rear-view mirror. Michael Portillo's fond notion of joining free-market economics with social liberalism may have a certain intellectual charm, but - ironically, given Portillo's past as a man of the hard right - it is a decidedly bien-pensant vision, in which modernisation can mean only the continued advance of liberal values. Outside the wine bars and think-tanks of Westminster, this view has little resonance. In contrast, a policy agenda that combines leftish-sounding promises on pensions and university tuition fees with a draconian stance on crime and asylum-seekers could be horribly effective.
If, as seems likely, Howard refines the policies fumblingly initiated under Duncan Smith, he could achieve something that new Labour has signally failed to do, and develop a genuinely post-Thatcherite agenda. Unlike Portillo's rarefied vision, this will not be yet another variation on 19th-century individualism. It will be a version of populism embodying a much harsher modernity, in which the power of the state is central.
Many will find this new Tory populism repugnant. Social democrats and Thatcherite true believers alike will be scandalised by this mix of old-left attitudes on public services with social authoritarianism. It is an ugly brew, but anyone who underestimates its potency is making a big mistake. In the chaotic environment produced by accelerating globalisation, right-wing populism has every prospect of thriving. Howard's version could enable him to solve the crucial conundrum that has defeated his predecessors - how to attract support among voters at large while retaining that of party loyalists.
Only a few years ago, credulous commentators were talking of a new Gladstonian era of peace and free trade. In a sense, the globalised world that existed before the First World War has returned - but it is far removed from the idealised version propagated by gushing Blairites. Like the late 19th century, the early 21st century is a time of geopolitical conflict and colonial wars. Control of scarce natural resources is a prime objective of the great powers, and the Great Game is being played again in Central Asia and the Gulf.
Far from democracy spreading everywhere, new varieties of popular authoritarianism are emerging. In Russia, where the triumph of capitalism was supposed to ensure democracy, a reversion to authoritarianism is under way. Vladimir Putin appears to have concluded that the country is not best governed by big business. This may discomfort some foreign investors, but he is likely to continue to be far more popular than Boris Yeltsin - the corrupt darling of western free-marketeers - ever was.
Advancing globalisation is producing a powerful political backlash - not in the confused and ineffectual anti-capitalist movement, but in the mainstream parties. Protectionism is on the rise, notably in the US, where both parties are busy reviving the Yellow Peril. In Europe, immigration has become the central political issue in almost every country, with all parties promising to do anything necessary to stem the flow of asylum-seekers.
Predictably, the political response to the dislocating effects of globalisation is a growing demand for strong government. People are turning to the state not as a guarantor of market choice, but as a buffer against insecurity. Now, as in the past, economic modernisation and an activist, often authoritarian, state go together. In the late 19th century, it was the Continental European right that first understood this truth. In Britain, more than a hundred years later, it may once again be the right that refashions the modern state. If so, it will not be Gladstone but Bismarck who supplies the model.
New Labour embraced Thatcherite economic policy as a badge of respectability, but it is now a liability. If, as seems certain so long as Blair remains leader, the government clings to the orthodoxies of the 1980s, the political initiative will shift back to the Tories. Like the Thatcherites it has so assiduously copied, new Labour will end as one of history's casualties.
The difficulties facing Howard's leadership are well known. The Conservative Party in the country is old, with an average age in the mid-sixties. Having frightened off donations from big business with its shambolic performance and strident anti-Europeanism, its finances are not in the best of shapes. With the first-pass-the-post system now tilted heavily against it, it has an electoral mountain to climb if it is to regain power. Contrary to the widely held opinion of liberal commentators, Howard's reputation as an illiberal home secretary may be an asset to him; but it is hard to see how he could go further in that direction than David Blunkett has gone. Above all, if Howard is crowned the new Tory leader, he must reconcile his party to the need to reach out beyond itself to the centre ground in the country.
This may be a daunting task, but I would not lightly dismiss Howard's chances of pulling it off. It is easy to indulge in wishful thinking about the great British public. True, in some respects, it is notably more tolerant than its leaders. An easygoing attitude to sex is now part of the national character. British culture is lax and hedonistic; it is weaned on debt and soft drugs, and has no memory of Victorian values. But that does not make it liberal. Most voters see no reason why the government should interfere with their pleasures, but they are unforgiving in their attitudes to those they see as law-breakers and outsiders.
Equally, the British public is not now - if it ever was - notably social democratic in its values. It has no time for Thatcherite dogmas on privatisation and it resents Blair's Thatcherite policies on higher education, pensions and, perhaps, the NHS. But it is not at all keen on high taxes, and has no fondness for redistribution when it does not benefit the affluent majority. Unlike Thatcher, the public is convinced there is something called society; but by society it means people like itself.
Any party leader seeking to garner the floating vote must be attuned to a mix of attitudes that does not fit easily into existing political categories. It is "liberal" on sex and drugs, "reactionary" on immigration and law and order, old Labourish in its view of some public services but grudging in its attitude to the taxes that are needed to pay for them. It may not add up to a pretty picture, but if there is a centre ground in politics, this is it.
The advantage of Howard's populism is that it can appeal to this British majority and at the same time be accepted by the Tory heartlands. It has become fashionable to view Hague with something approaching nostalgia, and certainly by comparison with Duncan Smith he was a leader of genius; but he left the Tories with a deeply damaging form of party democracy. If the procedures he bequeathed for the selection of a party leader succeeded in their immediate goal of denying that prize to Kenneth Clarke, they also enfranchised the most backward-looking section of the party. In so doing, they built into its structure a disabling alienation from the country that Britain has become.
Like his predecessors, Howard must placate the constituencies. His insistence that they be given an opportunity to ratify his emergence as leader shows he understands this fully, but if there is danger for him it is unlikely to come from that quarter. The populist modernisation he seems set to impose on his party is very much in tune with the interests and prejudices of its most benighted members. Like most voters, Tory backwoodsmen and women want higher pensions and heavily subsidised university places for their offspring. They will welcome the shift away from Thatcherism that these new policies involve.
The danger to Howard comes more from his parliamentary colleagues, who will find it a wrench to break with the habits of treachery they have formed over many years. The effect of toppling Thatcher was to inject a culture of disloyalty into the Tory party. It is now deeply embedded and may be impossible to uproot.
Still, Howard has a real, perhaps unrepeatable, opportunity to give the Tories another lease of life. His brand of populism seems well suited to a political culture in which success depends on flattering the electorate while mobilising its baser instincts. Even if, in the end, he fails, he will have made the best of a bad job.