If you go into any large American bookshop and study the contents of the self-help shelves, you are likely to be able to group the titles on offer into two broad categories. First, the books that tell you how to make a lot of money quickly. And, second, the books on how to cope with low self-esteem.

This combination isn't a coincidence: the two genres seem like necessary, almost inevitable bedfellows. Societies which insist that opportunities for success are unlimited and universal consign unwittingly their less successful members to feeling that they have only themselves to blame for their failures. Those at the bottom of so-called meritocratic, opportunity-filled societies end up not only poor, but also ashamed.

One of the more enduring paradoxes of modern life is how societies that are richer than ever before could have failed so dismally at the business of being happier than ever before. How can we have so much, yet still feel so lacking?

A possible answer lies in the psychology behind the way we decide what is enough. Our sense of an appropriate limit to achievement and wealth is never decided independently. It is decided by comparing our condition with that of a reference group, with that of people we consider to be our equals. We cannot appreciate what we have in isolation; neither can we judge it compared to the lives of our medieval forebears. We cannot be impressed by how prosperous we are in historical terms. We will take ourselves to be successful only when we have as much as, or more than, the people we grew up with, work alongside, have as friends or identify with in the public realm.

The rigid hierarchical system in place in almost every western society until the 18th century had denied all hope of social movement except in rare cases. It was unjust in a thousand all too obvious ways, but it offered those on the lowest rungs one notable freedom: the freedom not to have to take the achievements of quite so many people in society as reference points - and find themselves severely wanting in status and importance as a result.

It was a freedom, because it was always unlikely that one would ever reach the pinnacle of society. It is perhaps as unlikely that we could today become as successful as Bill Gates as that, in the 17th century, we could have become as powerful as Louis XIV. Unfortunately, though, it no longer feels unlikely - depending on the magazines one reads, it can in fact seem absurd that one has not already managed to find a business idea to revolutionise global trade.

One of the few ambitions shared by politicians across the party-political spectrum is that of creating a fully meritocratic society; that is, a society in which all those who make it to the top do so only because of their talents and abilities (rather than because of unfair privilege - upper-class parents, a friendship with the boss and so on). This meritocratic ideal has naturally brought opportunity to millions.

Gifted and intelligent individuals, who for centuries were held down within an immobile, caste-like hierarchy, are now free to express their talents on a more or less level playing field. But there is, inevitably, a darker side to this idea of meritocracy: for if we truly believe that we have created (or could even one day create) a world where the successful truly merited all their success, it necessarily follows that we have to hold the failures exclusively responsible for their lack of success.

In a meritocratic age, an element of justice enters into the distribution of wealth, but also into the distribution of poverty. Failure comes to seem not merely regrettable, but also deserved.

There are two ways to make people feel better about themselves: to give them more money or to restrain their desires. Modern societies have succeeded spectacularly at the first; but, by continuously in- flaming appetites, they have at the same time helped to negate a share of their most impressive achievements. The most effective way to recover self-esteem may not be to try to make more money or win more fame. It can be to distance ourselves - practically and emotionally - from anyone we consider to be our equal and who has become more successful than we have.

The price we have paid for expecting to be so much more than our ancestors is the permanent feeling that we are far from being all we might be.

Alain de Botton is the author of Status Anxiety (Penguin)