Choice is this political season's black. The newly crowned Michael Howard trumpets plans for education and health vouchers - giving parents and patients maximum choice. Tony Blair and Gordon Brown squabble about how hard to push the agenda for "consumer choice" in public services. Politicians are falling over each other to offer a smorgasbord of life decisions.
"Choice" is such an unquestioned good that the word itself is now an approving adjective. In all aspects of life, from pensions to life partners, from hospitals to handbags, the accepted wisdom is that More Choice is Better. In fact, the modern multiplication of options is at best a mixed blessing, at worst a curse.
In television, the explosion in channel choice has resulted, in the words of Roger Mosey, head of BBC Television news, in a "poisonous cocktail" of reality TV, smut and confessionals. And the vast array of financial products now available has done nothing to lift pension savings.
There is growing evidence that it is hard for people to deal with too much choice. A number of studies suggest that while we are able to make sensible decisions between, say, six options, being faced with 30 causes us to back off or "choose" entirely arbitrarily.
"As the number of available choices increases, as it has in our consumer culture, the autonomy, control and liberation this variety brings is powerful and positive," says Barry Schwartz, author of the forthcoming The Paradox of Choice: why more is less. "But as the number of choices keeps growing, negative aspects of having a multitude of options begin to appear. As choices grow further, the negatives escalate until we become overloaded. At this point, choice no longer liberates, but debilitates. It might even be said to tyrannise."
Schwartz cites studies showing that people presented with fewer options are more likely to take one of them - from exotic jams to extra-credit essays - and also to feel more satisfied with their selection, than those offered a wider range of choices. Markets are very good at generating ever more finely grained commodity choices - but it seems they are subject to diminishing returns. Financial services companies have learnt that offering hundreds of options is a disastrous ploy, unless they also suggest about half a dozen "recommended" or "favourite" products. Restaurants have always known this. If the over- abundance of choice were restricted to jam and tea, there would be little to worry about. But across the political spectrum, there is a growing desire to inject more choice into public services, on the largely unquestioned assumption that more choice is an unalloyed good. The choice fetish threatens our public realm.
Increased choice poses three fundamental challenges. First, too many options result in disorientation and regret. Second, most people, left to their own devices, systematically make bad choices in some important areas of life. Third, the spawning of multitudes of market-mediated choices often obscures the larger choices confronting us.
It seems odd, in a market society, to suggest that three choices might be better than 33. But individuals simply do not have the time, energy or cognitive skills to weigh the pros and cons of an ocean of possibilities before coming to a perfectly rational conclusion that will maximise their own "utility". And, in fact, trying to do so is a certain route to misery: people who are "satisficers", accepting a good enough option, are much happier than "maximisers", who scour the pages of Which? magazine and dozens of websites in search of the ideal product.
This is partly because choice comes with an unwelcome sibling: regret. If you choose Milan for a holiday rather than Barcelona, you necessarily give up what good things Barcelona has to offer - what economists call an "opportunity cost". But in a situation where the choices are limitless - a weekend in any city in the world, perhaps - the chances of feeling that another choice might have been better increase significantly. The more grass there is, the higher the chance of some of it looking greener.
This is as true of lifestyles as of holidays. When a group of friends includes some married parents, some swinging singles, a gay couple and a childless career couple, the odds on one or more looking at one of the others and thinking "you lucky sods" are high indeed.
"The 'success' of modernity turns out to be bitter-sweet, and everywhere we look, it appears that a significant contributing factor is the over-abundance of choice," says Schwartz. "Having too many choices produces psychological distress, especially when combined with the desire to have the best of everything - to maximise."
And while we rightly value our freedom to make our own decisions about jobs, partners, home towns and hobbies, the downside of autonomy is having to take responsibility for failure as well as success.
This, too, contains the seeds of regret, as Schwartz suggests: "When there are many options, the chances increase that there is a really good one out there, and you feel that you ought to be able to find it. When the option you actually settle on proves disappointing, you regret not having chosen more wisely."
In public services, this responsibility weighs even more heavily because the stakes are so much higher. If you choose a hospital for your mother's hip replacement, how do you feel if the operation goes horribly wrong? If everyone's child has to go to the local school, your son's failure to pass his GCSEs may well provoke anger at the school and/or him; but at least you are spared having to blame yourself.
The dangers of regret would be negligible if people typically chose wisely. But the second challenge to the ascendancy of choice is that in a number of situations, people choose badly. As a rule, we are lousy fortune-tellers; we are also very bad at learning from our mistakes. In the growing field of decision-making psychology and behavioural economics, a great deal of work has been done by Daniel Kahneman, the first psychologist to win the Nobel prize in economics, Daniel Gilbert, Tim Wilson, George Loewenstein and others on the gap between "decision utility" (I'll buy this vintage sports car and it will make me happier) and "experienced utility" (I've bought that vintage sports car and feel no happier at all).
Why the gap? Why do I wrongly predict future utility? The main reason is that I fail to anticipate adaptation: I don't think that I'll ever just get used to having such a splendid car, but I quickly do. And the intriguing part is that we don't appear to learn our lesson the next time. By the time the positive blip from the car has worn off, I'm after a helicopter. Gilbert and Wilson call this "miswanting".
These insights help to explain why the dramatic increases in consumption in affluent nations have done nothing to lift our collective mood. But there are other circumstances in which we cannot be expected to choose wisely - not least when we are very ill. The notion of choice in secondary healthcare is largely illusory, as Gordon Brown pointed out in his Social Market Foundation lecture this year. Half of all NHS admissions and three-quarters of all beds result from emergency, urgent or maternity cases which, as Brown says, are ones "where patients are generally unable to shop around" - although even he goes on to attack those who "devalue or ignore the important issue of greater consumer choice" in healthcare.
Professor Julian Le Grand, who is now advising the Prime Minister on healthcare reform, writes in his new book, Motivation, Agency and Public Policy, that making sensible choices about healthcare is difficult for several reasons: lack of technical knowledge; weakness of will; being in a highly emotional state; or lacking experience of the options on offer. And it seems that most of us are happy to delegate authority to medical experts, once we are faced with a real health problem. Research by the US surgeon and writer Atul Gawande shows that although two out of three people say they would want to choose their own treatment if they got cancer, among cancer patients themselves, only 12 per cent want to do so. The first figure shows the salience of the right to choose; the second demonstrates its limits in real life.
Another enemy of rational choice is the tendency of individuals to display, in economic jargon, a "high personal rate of time preference" - in other words, to live for today. A consistent problem in public policy is the unwillingness of young people to save anything like enough for their retirement. Given the astonishing increases in life expectancy, this should not be surprising: how on earth is the time-span over which we can think rationally supposed to keep up?
And yet, here as elsewhere, the solution is seen to be individual choice through the market. Arguing last year against greater compulsion in pensions and savings, Andrew Smith, the Secretary of State for Work and Pensions, asked: "How is the state going to be better at deciding what the future level of savings should be than the amalgamation of all the decisions made by individuals, their families and their employers?" Given that the amalgamation of those decisions is not terribly impressive, it should not be a radical thought that the state could indeed do better.
And Le Grand supports compulsory savings. But not on the grounds that people are foolishly short-termist, but because your "future self" - to whom your current self is only loosely attached - is not present at the decision-making point.
This, he says, opens up the "possibility of market failure. For there is now a group of people who are not participating in the market but who are affected by the decisions made by those who are participating in it." Roger Scruton elegantly describes this as our present free-riding on our future.
But Le Grand's argument is highly dubious. Once non-present selves get in on the act, there's no telling where we will end up. What about poor children in the Sudan affected by our agricultural consumption patterns? Or my unborn great-grandchild, affected by my burning of fossil fuels? Following Le Grand's logic, it is entirely rational to smoke heavily so long as it is pleasurable to your "current" self - after all, it is only your "future self" who will get killed by cancer.
What this demonstrates is that much intellectual ingenuity goes into the avoidance of arguing for a restriction of people's choices. The pensions case does not need Le Grand's contortions: people need to be forced to give up some present-day consumption choices in order to ensure that they have more of such choices in the future, because they cannot - and cannot be expected - to make such a choice for themselves.
Ordinary human beings are a very long way from the free-market economists who think us possessed of a brain like that of Pierre Simon de Laplace's super-intelligence, which could "comprehend all the forces by which nature is animated . . . an intelligence sufficiently vast to submit these data to analysis . . . nothing would be uncertain and the future, as the past, would be present to its eyes".
The third progressive challenge to the orthodoxy of choice is aimed at the prefix "consumer". In public discourse, choice is presented as being about putting power in the hands of the customer, who is king, while producers are subjects. Brown himself argues that it is wrong to "confuse the public interest with producer interests".
What is forgotten in the rush to consumer sovereignty is that producers are people, too - indeed, the same people who are consumers. Serving consumers' interests may undermine the interests of the people working to produce the goods or services they require. Phone lines open all day are terrific for me, but less wonderful for the staff operating them. My cheap kitchen may come at the price of terrible labour relations in the factory.
There are enormous dangers in the misuse of producer power, and the Labour Party has to be careful not to be associated with "vested interests". But equally, there is a real danger of automatically promoting consumer choice over worker well-being. This is especially significant given that an increased capacity to consume appears to have no impact on well-being, while a better quality of working life has a significant positive effect. The problem is that there are no mechanisms through which the choice between the welfare of worker and consumer can be made: the market has to favour the consumer, for she is the source of all profits. As the political scientist Robert Lane points out: "People do not buy economic systems or work cultures, they inherit them."
Choice is limited to shades of grey within certain demarcated areas. Markets are very good at providing an array of choice when it comes to cornflakes or hair conditioner. But they are lousy at presenting the big choices, such as a more preventative healthcare system, better working conditions or new transport solutions.
We are thus swamped with micro-choices which serve to obscure the larger macro-choices that lie beneath them. We are faced with a tyranny of small decisions; choices that turn out to be double-edged swords.
The proper arena for macro-choices is the political one. It is through political discourse that we choose to curb con- sumer power in favour of workers' rights, or vice versa; to grant patients buying power in the health service, rather than shifting to preventative medicine; to weigh the merits of compulsion in pension savings. But the orthodoxies of the market, consumer sovereignty and choice unite rather than divide our political leaders. Most arguments across the despatch box are between positions just inches apart on the political spectrum.
In order to address some of the chal-lenges of affluence, we urgently need a richer, more diverse political conversation. It is a savage irony that the politicians who proclaim the virtues of choice offer such a narrow menu themselves.
Richard Reeves is the author of The Politics of Happiness, a New Economics Foundation discussion paper