Just seven months ago, Labour suffered a defeat of epic proportions. It was its worst performance since 1918, barring the loss it suffered under Michael Foot in 1983. But where is the debate, the soul-searching, the way out of such a setback? The leadership election, and now the severity of the cuts, mask the scale of the defeat. Labour appears to be sleepwalking away from the car crash that was the 2010 election, but we know that this was more than just another turn of the electoral wheel. In May 2010, Labour lost more than an election: it lost a way of being.

To understand the full impact of the loss, we must first acknowledge that social democracy has been in retreat for years. The dominant centre-left model of political change was found wanting as long ago as the end of the 1970s. Profound shifts took place in the final two decades of the 20th century. Capital decisively globalised and financialised, fatally weakening the power of governments to regulate it and of trade unions to act as a countervailing force. Inequality intensified, our culture was commodified and class identities faded. The cold war - which allowed social democracy to nestle between command and capitalist economies - thawed out.

The biggest shift, however, was cultural. The way we now communicate and interact has had profound implications that the old social democrats never even began to grasp. These changes atomised society, but also offered new opportunities to create communities and foster collective action. But one thing is incontestable: the appetite for voice and influence stirred by "new technology" makes the top-down politics of the past century a non-starter.

In the face of such momentous changes, some on the left have clung to the old model of centralised statism. This has made life easier for the more zealous elements of New Labour, who have at least recognised the epochal developments still taking shape - though the New Labour project in government amounted to an elaborate attempt to paper over the cracks and to pretend that the centre left could adapt itself to these changes, rather than shape them according to a politics of modernised social democracy. The illusion that the "Third Way" had any meaning at all was shattered by the financial crisis, as was the idea that irresponsible capitalism could deliver the stability and resources required for incremental social improvement.

New Labour stayed in office for 13 years because the world economy was so strong and the Tories were so weak. But even in such benign circumstances, the poor got poorer and the planet burned. New Labour certainly attempted to ameliorate some of the symptoms of gross inequality, but it never properly tackled the effects of untrammelled markets. In fact, doing so was ruled out by the very foundations of its thinking: the assumption that this is and will remain a Conservative country, and that there is no alternative to a capitalism of unrestrained markets in which the sphere of non-economic life is squeezed by more production and consumption. The only plan they had was to stoke a finance-driven, lightly regulated economy, and then surreptitiously take the tax skim to fund social programmes.

Hope squandered

Yet it was clear even before the financial crash wrecked it that this model was severely flawed. It never talked up the morality of redistribution, and so was doomed to be the politics of ever-decreasing circles. Damning proof of the project's failure was contained in the British Social Attitudes survey of January 2010: among other findings, it showed that, in 1994, 51 per cent of the population believed the government should create a more equal society; by 2010, the figure had fallen to only 38 per cent.

New Labour lost five million votes between 1997 and 2010, four million of them under Tony Blair's leadership. Along the way, it squandered vast sums of goodwill and hope. By 2005, many in the broad Labour coalition were beginning to question what the party stood for. By the time of this year's election, the game was up - the party lost as New Labour.

Not that you'd have known it from listening to the brief flurry of post-election chatter from the Labour right. Alan Milburn, Giles Radice and the pollster and writer Peter Kellner all sounded off (the latter using an article in the New Statesman to argue that the party's only hope was to move further right), but their interventions fell flat. And small wonder: this school of thought would have us believe that any deviation from the politics minted by the last leader but one will condemn Labour to a generation in the wilderness. But it was exactly the refusal to deviate that led to the party's crisis in the first place.

It is as if New Labour and Old Labour have fought each other to a standstill. Both still have valid points to make, but both have failed electorally, and neither has any ideas about where to go next. Old Labour failed to respond to the new world; New Labour embraced that new world far too uncritically.

Labour recoils from debate because it has ­almost forgotten how to think, or even why it should. As ever, its default position is that socialism remains what Labour governments do. The contradictions duly pile up, and the challenge of unpicking them is so daunting that the party seems to have switched off.

So where is the light? It comes from two places: from leaders and from people, who tell us that it is both feasible and desirable to renew social democracy - socialism, even - but that renewal must be truly transformative. It cannot be about a change of direction, but must involve a paradigm shift to an entirely new form of left politics.

This new socialism has been cohering among writers, thinkers and activists for the best part of five years, though it is only just beginning to have the same effect at the very top of the party. Ed Miliband seems instinctively to understand this burgeoning branch, which is rooted in the idea of the good society and a recognition of the reforms of both state and market needed to make it a reality. He has shown a determination to talk about things no recent Labour leader has gone near: the idea of "life beyond the bottom line", "the things that business cannot provide" and the dangers of "naivety" about markets. Now, he has three options: he can turn round and go back, he can triangulate between Old Labour and New, or he can keep going.

The leadership should take heart and confidence from underlying trends in Britain. The project of creating the "good society" starts where people are: stressed, stretched, anxious, insecure, tired and alienated. This realisation has bubbled through our politics for a few years; now it is properly taking shape. Today, when Ed Miliband talks about the "squeezed middle", he is on to something - though the same pressures and constraints apply even more to people towards the bottom.

This common condition, defined as a loss of control over key aspects of our lives, crystallises the challenge facing the new socialists: whereas New Labour tried to bend people's aspirations to its resigned and deflated world-view, the new paradigm seeks to grasp their hopes and fears, and deal not with the symptoms of our social and economic recession, but the causes. We think that what is starting to take shape should be called New Socialism. Here are its salient features.

Beyond the market

First, although it emphasises greater equality of outcome, New Socialism understands that we now have to focus on the non-material things that foster contentment and fulfilment. We have to place much greater value on time, care and co-operation, and build a culture of belonging founded on something deeper than just production or consumption. We have to redefine "aspiration" to bring it into line with people's real hopes: not just to earn and own, but to reach their full potential and live in a society that is safe, caring and neighbourly.

Second, in the new paradigm, capitalism is understood as a dynamic force that is at once creative and destructive. Unless it is shaped and regulated, it undermines society and com­munity and, crucially, its own best self, as the trailblazing entrepreneur is sidelined by the monopoly corporation and the thriving town centre by the out-of-town mall. If we don't regulate free markets, we end up regulating people, forcing them to behave as the market desires.

New regulation will have to happen at a transnational level, through the EU at the very least. Economic sovereignty will have to be pooled to ensure that the damaging effects of capitalism are tamed. Every second of every day, the bond market pools its sovereignty to destroy whole economies. To match such power, there should be moves towards controls on speculative capital flows, co-ordinated corporation taxes and the establishment of the principle of a Europe-wide minimum wage.

Third, New Socialism knows the state is vital, but recognises, too, the crisis of the bureaucratic and market state. It wants a state whose scope
is determined democratically and that is made accountable, responsive and local through the boldest political reforms of public service this country has ever seen. A more proportional electoral system is only one part of the change required: the state must be reinvented so as to entrench citizens' involvement through the principles of democracy and co-production. Parents expect meaningful input into their children's education; patients increasingly want their treatment to be based on dialogue; the people who work in the public services can contribute far more than the implementation of diktats. Health, education, social care and much more need to be liberated from the bureaucratic, outsourced state and reshaped collectively and democratically.

Fourth, New Socialism is determined to get to grips with the most pressing issue of our age: climate change. This is the toughest challenge of all. It involves the reconciliation of social justice with sustainability, and requires that we question the relentless pursuit of growth - hence, in part, the critical role of a post-material agenda. Social democracy has to cease being solely about sharing out the proceeds of ever-increasing material wealth. Instead, it needs to offer a vision of the redistribution of time, power and well-being.

Finally, this New Socialism recognises that Labour must end the practice of Labourism. This is the view that "socialism" can be delivered from the centre by one - and only one - all-seeing, all-powerful, monolithic party. Today, we live in an era of pluralism, with competing centres of power. This new politics is manifested most clearly in Westminster in the form of the coalition, but can also be seen in Edinburgh, Cardiff, Belfast and on councils the length and breadth of the nation. This cultural shift is huge and, for Labour, unavoidable. The party has to redemocratise and, at the same time, engage with a range of different parties' forces and voices.

A realigned and revived centre left is held together by the good society: the only vision that holds out convincing answers to issues of equality, esteem, time, sustainability and quality of life. Unless and until socialism feels like a better offer than the treadmill of consumerist capitalism, there is no hope. We cannot build the good society on the rotting corpses of bloated bureaucracies and failed markets.

The new paradigm shares New Labour's desire to win, but wants to win for a radical purpose. It offers a self-reinforcing rather than a self-defeating politics, because it deals with the causes of the economic, social and environmental crises we face, not just the symptoms. As such, it seeks to break the mould of British politics. Its origins can be traced to the possibilities that abounded in the early days of New Labour, but it also looks for inspiration from Greens and social liberals, from civil society groups such as London Citizens and the Facebook generation. It even finds echoes in David Cameron's "big society", though it sees the many flaws in that model. And it might just have a Labour leader who understands it.

Leap of faith

Ed Miliband has to make the leap to this paradigm. But, then again, why would an ambitious 40-year-old struggle to win his party's leadership, having to beat his brother in the process, only to replay the miserable, soul-destroying period he has just gone through?

That's his choice and ours: either we redefine the centre, create a new common sense - and therefore a new political paradigm - or we reconcile ourselves to the way Britain has been defined by the right, and face the steady erosion of all that we hold dear: party, movement, state, civil society, unions, democracy, sustainability, co-operation, care and time.

The walls are closing in on the old social democracy. In the last round of elections in mainland Europe, the mainstream parties of the centre left fared dismally, managing 30 per cent of the poll in Sweden, 29 per cent in the UK, and only 23 per cent in Germany. Everywhere the old social democracy is dying. Attempts, such as New Labour's, to revive it rather than transform it have failed.

There is precedent for the idea of a paradigm shift. Historians and theorists such as Arthur Schlesinger and Albert Hirschman observed that, roughly every 30 years, society shifts - from the public to the private and back again. After a while, the grass looks greener on the other side. The late 1940s to the late 1970s was the era of the public, the late 1970s until now, the age of the private. Today the conditions are right for another turn: to a new common life, and the security and freedom this affords - but only if we make it happen by tackling a market that is too free and a state that is too remote.

Centre-left politics has always been divided between optimists and pessimists, between people who attend only to the symptoms of inequality, sustainability and political dysfunction, and those who bravely choose to deal with causes. These are the people who make history. In the midst of one of Labour's most trying periods in decades, this is the offer made by New Socialism: the chance of an audacious reinvention, in an age that needs it more than ever.

Neal Lawson is chair of Compass. John Harris writes for the Guardian.