Boris Johnson is Britain’s most popular politician. The competition isn’t even close. David Cameron is facing not just mid-term blues, but the additional conundrum of how to nurse a fragile Coalition through them. While Ed Miliband looks to define himself in a way that might start to break through into the public consciousness; George Osborne and Nick Clegg are vying for the wooden spoon in the unpopularity stakes. So Boris stands alone, through an astounding gravity-defying feat. Only he can get stranded on a zip wire and come out looking good.

All of the standard anti-political complaints about the entire political class – “they are all the same”, “they have never had a proper job”, and “nobody seems to stand for anything anymore” – all come with an asterisk, and an “except for Boris” disclaimer. That is no mean feat. But his opponents still don’t get Boris. Instead, he is often treated as one big joke, with his popularity explained away by casting him as Westminster’s court jester. Indeed, his opponents risk showing some contempt for the voters by rather patronizingly suggesting that the public are just having a laugh, and will soon need to sober up and get serious about politics again. It’s about time they stopped laughing and started learning.

The tendency to dismiss Boris can be found across the political and media spectrum, among MPs and commentators on left and right. It may, though, be particularly dangerous for Labour and the liberal-left, who risk repeating the mistake of fatally underestimating political opponents that they have made too often before.

Misunderstanding the appeal of Boris fits a pattern through which rational liberals demonstrate a disconnection from how people think about politics and the role of personality within it. How many people were adamant that Ronald Reagan, perhaps the politician who Boris Johnson resembles most, could never make it from Hollywood to the White House?

Remember, more recently, the widespread assumption that Al Gore, as a brain-box policy wonk, had only to turn up to the televised Presidential debates for George W Bush’s campaign to collapse into ridicule?

And Boris’ fellow blond Margaret Thatcher was also deeply underestimated on her road to Downing Street. Partly that reflected discombobulation at the sheer novelty of a woman bidding for the Premiership in the 1970s. But she was also dismissed as a Home Counties politician, unlikely to play outside the south, just as Boris is assumed to be a local London brand.

Boris Johnson has been extremely lucky in how his political opponents treated his bid for power in London, being just as “misunderestimated” by his political opponents as Bush and Reagan were. In London, Labour cast him as a clown and could not believe the voters would ever elect him, right up to the moment that they did. By helping to set expectations so low, implying that Boris could never be up to the job, and that it would be a miracle if London survived his mayorality, Labour massively boosted his chances of re-election. If they don’t change their tune, they could end up letting him walk into Downing Street without ever offering him any serious scrutiny.

People like Boris because he is real. Of course, Boris brings an element of performance to “being Boris” – a splendid character homage to PG Wodehouse – but the persona is inimitable, because it is a self-evidently authentic reflection of the man who he is. People want authenticity and character in politics, but struggle to find it from what they perceive to be an identikit political class.

So people like Boris because he is unscripted. He takes risks. He writes what he thinks in his newspaper columns. And it helps that the London mayoral system means he is the one national politician whose position does not depend on patronage.

The left also believes that Boris will fail because he is too right-wing. Guardian columnist Polly Toynbee wrote recently about his “dog whistle” to the right. This charge of “dog whistling” is much over-used by the left. And it is a particularly odd claim about a man who will headline a fringe meeting/pro-Boris rally called “Re-elected and Olympotastic” at the Tory conference. Boris tells everybody what he thinks using a foghorn, not a dog whistle, using his Telegraph column to let his views be known well beyond his London bailiwick. Where Boris is right-wing, he is out and proud.

But he is among the most liberal of Tories too. People also like Boris because he is an optimist. Boris doesn’t have to learn to love Danny Boyle’s Britain, because he naturally does. Like Reagan and Thatcher, and indeed Blair too, Boris Johnson can stake a claim to the future, because he believes in it, and wants to live in it. Being Boris is entirely incompatible with the narrative of cultural pessimism which can leave the right marooned somewhere in the 19th century. That is why he described the idea of a “broken society” as “piffle”. On the night of the Olympic opening ceremony, Boris was telling a Guildhall audience about why the benefits of immigration in making London the world’s greatest city. While Tories worry about their inability to win ethnic minority votes, Boris won truckloads of them in London, winning over a million votes to return him as Mayor.

Strangely, the left is less comfortable with optimism than the Boris wing of the right. It may be that the inclusive patriotism of the Olympics offers an antidote to the allergic reaction which too many on the left still seem to respond to any form of national pride. But much of the left worries about moments which cheer the country up – fearing that it could reinforce the status quo, and take people’s mind off austerity and cuts. This is a deep strategic error. In eras when the left has sought to mobilise anger and despair, such as the 1930s and the 1980s, it has been routed. It has prospered only in those moments when it found a way to articulate national renewal and a sense of shared hope – in 1945, 1966, and 1997.  The instinct of many, struggling to define a new argument for the left, is to at least defend its great historic achievements of the Attlee post-war settlement. But the modern left will need to find its vision of the future too, as Ed Miliband’s policy chief Jon Cruddas acknowledges in his New Statesman essay this week.

So Boris’ ability to connect with optimism is something that the left could learn from if it did start to take Boris seriously, though it may well struggle to emulate the sprinkle of stardust with which he cheers people up.

Of course, the popularity of Boris may not last. Governing London is different to being given the nuclear codes. He is a politician who believes in soaraway ambitions, and may find it harder to connect to feelings of insecurity. That challenge mirrors the opposing test for those politicians on the left, who are in touch with insecurity, yet who struggle to articulate a more aspirational vision. In times like these, politicians need to connect to both hopes and fears.

There is no vacancy in the Tory leadership for some time yet.  Boris’ supporters may worry about his kicking for home a lap or two too soon. In Birmingham next week, Boris faces what David Miliband called a “Heseltine moment” when speculation about his challenging Gordon Brown dominated Labour’s 2008 conference. But Boris seems less likely to slip on a banana skin in handling the pressure.

Boris has benefitted, in City Hall, from being both in power but outside the Coalition. An oppositional persona is harder to maintain if facing the trade-offs in government – though both Reagan and Thatcher showed that it can be done.

Nobody knows if Boris can turn popularity into Downing Street power. But the real joke will be on anybody who is still laughing at what has surely become a serious prospect.

Sunder Katwala is the director of British Future, who will be holding fringe debates in Manchester and Birmingham over the next two weeks.