The financial crisis has been a turning point for the whole world. But it is not just decades-long prosperity that is under threat. So, too, are old ways of thinking. New concerns are stalking our world. How to address growing inequality? How to create wealth and reward it? How to ensure companies have good values as well as good products?

A new agenda for change is developing. This is as much a challenge for public policy as it is one for private corporations. Governments are recalibrating how they interact with businesses. Politicians are dusting down powers – of regulation, taxation and intervention – that they have long felt unable to use. When free markets fail as spectacularly as banking did, that response is understandable. The danger is that, in their zeal to correct under-regulation, governments end up stifling wealth creation and economic growth with over-regulation. Worrying signs of that are emerging.

There is another danger. The public and political desire to put right what went wrong in financial markets distracts governments from their even more fundamental task – of putting their own financial house in order. Not only has the global crisis put excess in the financial markets in the dock, but government excess and sector debt are on trial, too. Global government deficits hit $4trn by 2010.

There are arguments about whether short-term financial stimulus from governments can help or hinder economic recovery. There are debates about how much and how quickly public debts should be cut back. Britain is not alone in witnessing political disputes over how far and how fast deficit reduction programmes should be implemented. A similar discourse is taking place from America to Australia, Germany to Greece. What is not in dispute is that governments have to reduce their debts. However, taking short-term action in order to do so is one thing. Sustaining action so that governments in the longer term are better able to balance their books and rein in expenditure rather than fuel it with debt is quite another. There has been much focus on the first challenge – but too little on the second.

That must change. Governments across the developed world will have to develop more than a knee-jerk, short-term reaction to the fallout from the financial crisis. Ad hoc cuts in public spending will need to give way to a far more strategic reshaping of the way public services are delivered. A series of formidable long-term challenges will compel governments to change what they do and how they do it. The most significant of these is the profound change in demographic structures, which are placing huge strains on welfare systems. Already one-quarter of the population of Germany and Italy is elderly. Before long Britain and Spain will be the same. The older we get, the more we cost. In the UK we spend four times as much on health care for the over-eighties as we do on the over-fifties. Unsurprisingly pensions and health-care budgets have been growing unsustainably.

Now a crunch is coming. As the baby-boomer generation grows old, workforces and, therefore, tax revenues are sure to diminish. As the Office for Budget Responsibility has warned, even if the UK's structural deficit is eliminated in the short term, this will confront any future government with a major fiscal challenge in the medium term.

Something has to give. Demand for services is set to rise but revenues are set to fall, and the options for governments are limited. To date, they have looked to later retirement, longer working, higher taxes. Yet government spending is already consuming over 50 per cent of the eurozone's GDP and approaching 60 per cent in some Scandinavian countries. Meanwhile, there is growing public scepticism about the ability of governments to tax and spend wisely. Voters in Britain, Ireland, Portugal, Spain and Sweden have all opted for parties that promise restraint, not expansion. It is time to wake up and smell the coffee.

There is a clear message here. The truth is this: the era of big public spending is over. Austerity is the new normal. It is not a temporary phenomenon. It will become permanent. Fiscal conservatism is the order of the day. New rules on spending, debt and borrowing are emerging in Europe. And we will almost certainly see some countries introducing binding legal obligations and creating monitoring machinery to ensure that, in future, public spending growth is tied to economic growth.

The implications for what the state does and how it does it are profound. The old political arguments between the parties that have dominated recent elections in the UK about whether governments should spend a bit more or a bit less are already starting to give way to a more fundamental debate about the size and the role of the state. The implications for public spending of a more constrained fiscal environment are threefold. First, governments will be forced to choose between spending programmes as well as within them. Second, they will have to think in a new way about how to get better value from existing programmes. And third, Europe's largely tentative efforts to reform welfare and public-service provision will have to move up a gear.

None of this will be easy, yet it can be done. Take health care. In most developed countries health care, after pensions and benefits, is the biggest item on the public expenditure slate. For five decades now, health-care spending has outpaced economic growth in every OECD member state. In 1960 it consumed less than 4 per cent of GDP. Today it consumes almost 10 per cent. We have been spending more than we have been earning. You have only to look at what would happen if the trends of the past few decades continued to realise that is not a sustainable proposition. Within 30 years, the UK would be spending close to 15 per cent of its GDP on health care, France 20 per cent and the US 25 per cent.

Better, cheaper, faster

That is not to say that health care hasn't made great gains in recent times. It has. Deaths from cancer and heart disease have fallen sharply. Or that, for example, the investment that New Labour made in the NHS was wrong. It wasn't. A health system in which most buildings were erected before the NHS was created, where patients waited 18 months for hospital treatment and where shortages of doctors and nurses were routine clearly did not meet requirements. Under New Labour, NHS care improved hugely and this finds a ready reflection in the record levels of public confidence the service enjoys.

But health care is surely unique among modern industries, in that improvements in quality have not been matched by reductions in cost. Think of the price and quality of cars, computers or cellphones compared to a generation ago. Doing more with less and doing it better, cheaper and faster has become the norm. It is time for health care to catch up.

Stagnating resources and rising demands – diabetes rates worldwide are expected to rise by 50 per cent and the global cancer burden to double in the next two decades – will require governments, as the global competitiveness guru Professor Michael Porter puts it, to place a new focus on achieving high value for patients as the overarching goal of health-care delivery. The alternatives to that approach – rationing the care available to patients or forcing them to pay for more of it themselves – aside from being politically unappealing, would have negative implications for equity in health care. And if the only other alternative – to tax and spend more – is also off the agenda, there is only one game left in town: achieving better outcomes for lower costs. This is the new holy grail of health policy.

In the next decade and beyond, governments will need to think of themselves less as big spenders when it comes to health care and more as switch spenders, where resources are switched from less optimum services to others that produce better outcomes for lower costs. That will entail resources being switched from hospitals to community services and from treatment to prevention. And it will demand driving forward reforms that empower patients, financially incentivise outcomes, increase com­petition, improve transparency and devolve accountability to local care organisations. The overall direction of travel will need to be away from a top-down system to a bottom-up, where change is driven less by command-and-control and more by managed-market mechanisms in which power is moved from providers to patients. To be effective, however, these reforms will need to work in tandem with the old NHS mechanisms of strategic planning and priority-setting, which help bring coherence and equity to the health system. The change that is needed is not towards a market free-for-all – you have only to look to the US to see the impact on cost control where that happens – but towards an NHS where a judicious mix of levers is deployed to improve both efficiency and quality.

Making such changes will prove a challenge for the left as it will for the right. The question is who is better equipped to see through reforms that will often be controversial and sometimes unpopular. At first blush, reforms that call on market disciplines, customer focus and value creation would seem to put the right in pole position. Yet right-wing governments have failed to capitalise on this apparent advantage. In France, Nicolas Sarkozy's rhetoric promising a welfare and public-service revolution found only the faintest of echoes in wholly timid reforms. A similar timidity has characterised the right's approach to reform elsewhere, most notably in Germany and Italy.

In the UK, the coalition government's health reforms may have been more ambitious but they were badly misjudged. Radical reform requires a grown-up conversation with voters. Politicians need to be able to explain the rationale for change. To do that, they have to be trusted enough to be heard. The Conservatives failed to gauge how much political permission they had to make change. Working out what you want to do against what you can actually do is the essence of government. Good government is when policy and politics come together as one. Bad government is what the Health Secretary, Andrew Lansley, did when he unleashed his reforms.

Obsessed with policy tinkering, Lansley ignored the politically inconvenient truth that the Conservatives simply did not have enough public trust on the NHS to inflict change within it. The baggage they carried of being ideologically obsessed with privatisation weighed them down once they hit a wave of opposition to their health reforms. They are drowning as a result. The government's reforms are riddled with complexity and compromise. They fail to equip the NHS with the tools it needs to re-engineer itself for the new world of permanent austerity it is now entering. This failure has a profound political consequence for the Conservatives – they have forfeited any claim to be the party of NHS reform.

The left has a different problem. Labour has the permission to make change. The question is whether or not it has the volition. The NHS was created by Labour and it enshrines Labour values. The public trust Labour enjoys on the NHS is why, as health secretary in Tony Blair's government, I could make radical changes to it. These reforms were not necessarily popular at the time but, as always, the proof of the pudding is in the eating. The introduction of national standards, devolved decision-making, financial incentives, market mechanisms, private provision and patient choice drove major advances. Waiting times fell. Quality rose. Services improved.

Too often the left in Europe has shied away from such an approach. It has adopted a protectionist rather than a reformist approach to the public services. The left's default position has been to stand up for producers, not consumers. Defending the status quo in a world of such rapid change has proved to be a recipe for electoral disaster. In France, Germany, Italy, Spain, even Sweden, the left has suffered consecutive election defeats where until recently it could lay claim to be the natural party of government. As New Labour proved, it is not by being protectionist but by being reformist that the left is able to win.

Road to recovery

Opposing change might be comfortable, but it is not what is needed. Standing still in a world of rapid change is not a feasible option for the NHS. Nor is it feasible for the Labour Party.

Labour today has a big opportunity. The failure of the Conservatives' health changes leaves the reform terrain wide open. Ed Miliband should use the hiatus around the government's Health and Social Care Bill to set out a reform blueprint that can make the NHS sustainable for the long term. There is a similar opportunity in economic policy. He and Ed Balls were right to say that Labour could not promise to reverse every coalition cut to public spending. That was a necessary first step on Labour's road to recovering its economic credibility. The next steps are to lay out how a Labour government would balance the books. This should not just be a case of indicating which areas of public spending would rise and which would fall. It should also indicate how value – getting more out for what is being put in – would be improved across the public services. Clear commitments to major reforms of services will be instrumental to Labour regaining its reputation for econo­mic competence. This will be the central battleground of the next general election.

A new era of fiscal conservatism and radical reformism beckons. It is not just a more responsible form of capitalism that will emerge; it is a more responsible form of government. The big question for politics is whether it will be the right or the left that ushers it in.

Alan Milburn was secretary of state for health from 1999 to 2003