Progressive ideology relies on the capacity of human beings to live fulfilled lives in a just and co-operative society. That people whose beliefs imply optimism seem to spend most of their time wallowing in pessimism is one reason that leftists sometimes lack personal credibility (another reason being that egalitarians so clearly enjoy being very well-off). But miserable idealists need to make a New Year resolution to look on the bright side. Pessimism is becoming an impediment to progressive politics. It is 50 years since J K Galbraith coined the phrase "private affluence and public squalor"; today, the dichotomy is between private hubris and public pessimism.

It is pessimism of a particular and pernicious kind. People are not generally negative about their own lives. In fact, we systematically exaggerate the control we have as individuals. As Malcolm Gladwell, among others, has shown, we tend to give our conscious minds credit for many reactions that are in fact instinctive. Other studies - of what we say has made us happy and what has actually increased our levels of contentment - show that we have a huge capacity to rationalise our life choices. When we are forced to make a choice between limited options, we are as likely to end up claiming the choice as our own as we would if it were unconstrained. And the more we like a future possibility in our lives, the more inclined we are to believe it will happen. The human mind is hard-wired to be personally Panglossian.

In contrast, we are unduly negative about the wider world. As a government adviser, I would bemoan what we in Whitehall called the perception gap. Time and again, opinion polls expose a dramatic disparity between what people say about their personal experiences and about the state of things in general. Take attitudes towards public services. In a recent poll, 81 per cent of respondents said that they were happy with their last visit to hospital. Yet when the same people were asked whether they thought the National Health Service was providing a good service nationally, only 47 per cent felt able to declare it was so, and most think the NHS is going to get worse.

This perception gap is not restricted to public services, as a recent BBC poll on families confirms. Some 93 per cent of respondents des cribed themselves as optimistic about their own family life, up 4 per cent from the previous time the survey was conducted, 40 years ago. Yet more people - 70 per cent, across race, class and gender - believe families are becoming less successful overall. While we apparently thrive in our own families of many shapes and forms, as social commentators we prefer to look back, misty-eyed, to the gendered certainties of our grandparents' generation.

What is true for families is true for neighbourhoods: we think ours is improving while community life is declining elsewhere. We tend to like the people we know from different ethnic backgrounds but are less sure about such people in general. We think our own prospects look OK but society is going to the dogs.

The media seem to be the most obvious cause of this phenomenon. Bad news makes more compelling headlines than good. Tabloids and locals feed off crime stories, middlebrow papers are dismayed at the chaos of the modern world and the alleged venality and ignorance of those in power, and left-leaning broadsheets enjoy telling us that global instability is endemic and envir onmental apocalypse inevitable. Mean while, the content of television programmes - from dramas to news bulletins - contributes to what the communication theorist George Gerbner called "mean world syndrome": people who regularly watch TV systematically overstate the level of criminality in society.

Yet it is too easy to blame the media; the job of commissioning editors is to give us what we want. We make our own contribution to social pessimism. In the burgeoning industry of reputation management, it is generally argued that people are much more likely to tell others about bad experiences of services than good ones (5:1 is the usual ratio). Academic research suggests that people tend to exaggerate in the direction of the general mood. Viewing our own lives positively but wider society negatively, we will tend to pass on and exaggerate evidence that supports these prejudices.

Evolutionary determinists may seek an explanation of our predilection for bad news in neurological hard-wiring; perhaps, for the survival of hunter-gatherers, warning is more important than celebrating. But it is in two of the mega-trends of modernity that more likely reasons for our social pessimism are to be found.

First, there has been the inexorable rise in individualism since the Enlightenment. As Richard Sennett brilliantly argued in The Fall of Public Man, aspects of modernity such as the power of consumer capitalism and the ubiquity of the idioms of psychotherapy have accelerated the process by which we see our authentic selves as revealed in the private and personal spheres, rather than the public and social.

Unstoppable force

Hand in hand with the rise of individualism, we have seen the decline of industrial and pre-industrial collectivist institutions, including the organised church, trade unions, political parties and municipal elites. Robert Putnam's work on social capital suggests this decline in collectivism reaches down into our social lives, with people choosing to spend less time with acquaintances and more with intimates. Putnam's more recent work controversially argues that trust levels are lower and loose social networking less common in more diverse communities.

This points to the second of modernity's mega- trends. Increasingly, we feel that we are the victims of processes set in train by human activity but no longer under anyone's control. Globalisation is the gravity of modern society: an unstoppable force that will knock us over if we try to defy it. The origins of the current credit squeeze in the US sub-prime mortgage market show a financial system that is beyond not only its managers' control, but even their capacity to chart.

Illegal immigration, terrorism and pandemics are seen as the inevitable flip side of cheap travel and consumer goods. Philosophers and policy-makers argue about how best to regulate emerging science and technology in genetics, nano technology and artificial intelligence. But can anything long delay the advance of knowledge - especially if it has commercial applications?

It is not only that we as ordinary citizens feel beset by forces beyond our control. We are ever less likely to believe in the power or authority of our elected representatives (although we much prefer our own MP to MPs in general). At a time when they have more to prove to us than ever before, our leaders are diminished by the politics of a populist consumerism. In this time of uncertainty, is it surprising that the more politically successful national leaders - think Chávez or Putin - are those who offer strong leadership in defiance of democratic constraints?

This is the anatomy of social impotence. By definition, progressives argue for the possibilities of progress; but is anyone inclined to believe us? A hundred years ago, Joseph Rowntree established his charitable works after analysing the social evils of his age. When, last year, the Joseph Rowntree Foundation asked today's public for its definition of the "new social evils", the list had changed very little. Greed, poverty, crime, family and community breakdown all featured on both lists. But at a seminar to discuss the findings, advisers from the foundation and elsewhere agreed on one big shift between the late-Victorian era and today: while Rowntree had seen his evils as the unfinished business of society's onward march, today we see social patho logies as the inevitable consequences of an idea of progress that itself feels imposed upon us.

Brainier than before

And yet. There is a different story to be told about our world. It is a story of unprecedented affluence in the developed world and fast-falling poverty levels in the developing world; of more people in more places enjoying more freedom than ever before. It is a story of healthier lives and longer life expectancy (obesity may be a problem, but it is one that individuals have more chance of solving than rickets or polio). Think of how we thrive in the diversity of modern cities. Think, in our own country, of rivers and beaches cleaner than at any time since the Industrial Revolution. When you read the next report bemoaning falling standards in our schools, remember the overwhelming evidence that average IQs have risen sharply over recent decades. If you think we have less power over our lives, think of the internet, of enhanced rights at work and in law, or remember how it was to be a woman or black or gay 30 years ago.

As for the powerlessness of leaders, the Bali deal last month may leave much to be resolved, but isn't this at last a sign that nations can unite in the best interests of the planet? And should we really lose faith that human determination and ingenuity ultimately will win through? Despite the power of international finance, this is a world where it is possible to be economically successful in societies as deliberately different as those of Sweden or the United States.

We rightly worry about rogue states and terrorists with dirty bombs; but let us also remember that since Nagasaki we have managed to carry on for 60 years without anyone unleashing the power of nuclear warfare. Not only have there been three generations of peace in Europe, but when in the past has a project as grand as EU enlargement been accomplished, let alone accomplished in a decade?

Progressives want the world to be a better place. We bemoan its current inequities and oppression - yet if we fail to celebrate the progress that human beings have made, and if we sound as though the future is a fearful place, we belie our own philosophy. Instead, we need to address a deficit in social optimism that threatens the credibility of our core narrative.

There are many aspects to this; we should, for example, be making the case for a more balanced and ethical media. But my starting point is the need to forge a new collectivism. It is in working with others on a shared project of social advance that we can be reconnected to the sense of collective agency so missing from modern political discourse. It is the attitude of the spectator that induces pessimism, the experience of the participant that brings hope. The problem is not that change brings fear and disorientation (there's nothing new in this), it is that we lack the spaces and places where people can renew hope and develop solutions.

The old collectivism is dead or dying. Its characteristics - hierarchical, bureaucratic, paternalistic - are no longer suited to the challenges or the mood of the times. The institutions of the new collectivism must be devolved, pluralistic, egalitarian and, most of all, self-actualising.

For all the talk of the decline of social capital, people are doing more stuff together. Twenty-five years ago, with falling audiences, commentators assumed that the cinema and live football were dead: we would all rather stay in the safety and comfort of our new, hi-tech living rooms. But then the multiplex, the blockbuster, the all-seater stad ium and foreign players showed the problem to be no deeper than the failure to keep up with modern tastes and expectations.

Self-actualisation is the peak of Maslow's hierarchy of needs. There is evidence that more of us are trying to climb that hierarchy. It is in the crowds at book festivals and art galleries, in ever more demanding consumerism with an emphasis on the personal, sensual and adventurous. We want to enjoy ourselves, to be appreciated and to feel we are growing from the experience. Compare that to the last Labour Party, trade union or council meeting you went to.

Roll up your sleeves

The failure to provide routes to collective fulfilment means we assume that our journey is best pursued alone. In the 1970s and 1980s, new left movements at home and abroad placed emphasis on forms of political organisation and debate that were innovative, exciting and (dare I say it without mockery) consciousness-raising.

Today, there are signs of a yearning for new ways of working together. There is the growing interest in social and co-operative enterprise and the emergence of new forms of online collaboration. Gordon Brown's citizens' juries are a tentative step in the right direction, albeit without much fun or risk-taking, but generally, progressives seem more interested in bemoaning the state of the world than in rolling up their sleeves and getting to work on building the institutions of a new collectivism.

Despite the huge impersonal forces of the modern world, people are prepared not only to believe in a better future, but to work together to build it. Tackling climate change offers a fascinating opportunity to interweave stories of action at the individual, community, national and international levels. This potential will be fulfilled only when we provide spaces for collective decision-making and action that speak to the same vision of collaboration, creativity and human fulfilment that progressives claim to be our destiny.

Matthew Taylor is chief executive, Royal Society for the Encouragement of Arts, and former chief adviser on political strategy to Tony Blair