The attack on the World Trade Center and the British-American response to it have unleashed a wave of pessimism about the future of the "civilised world". No longer can we take our safety or our way of life for granted. We brace ourselves for anthrax in ventilation systems and nerve gas on the Tube. Security takes precedence over civil liberties. Race relations deteriorate. The economy heads towards recession. As we watch the rich nations of the world bomb the poorest, it is hard to avoid the conclusion that we belong to a civilisation in decline. And underneath it all, there is the uncomfortable feeling that the west has only itself to blame.
This kind of cultural pessimism has been a recurring feature of the intellectual history of the west. Indeed, before the 18th century, few in the west who thought about it ever had any doubt that their civilisation was declining. This was a legacy of both Christianity and the classical world. In Judaeo-Christian traditions, the fall from paradise and the coming apocalypse provided a kind of archetype of decline and destruction; and history as degeneration - or at best as repeating cycles - had been promoted by many classical scholars. Prophets of progress were hard to find.
Even during the 18th and 19th centuries, when post-Enlightenment doctrines of progress were at their height, powerful dissenting voices mocked or denounced the idea that civilisation might be advancing. The French philosopher Jean-Jacques Rousseau, for example, argued that modern civilisation displayed a level of corruption unknown in primitive societies. The Scottish historian Thomas Carlyle believed our "mechanical" way of thinking had made us morally inferior to other civilised ages. The German sociologist Max Weber foresaw a growing bureaucratisation of the human spirit. "Not summer's bloom lies ahead of us," he wrote, "but rather a polar night of icy darkness and hardness." Though Karl Marx had a quasi-religious faith in the triumph of communism, his vision of the capitalist world was profoundly pessimistic.
After the events of the first half of the 20th century - genocide, "total war", Stalinism, Hiroshima and so on - these voices sounded far more plausible than those that promised perpetual progress.
The pessimists of the 19th century were joined by many more in the 20th. Oswald Spengler published his magnum opus on the Decline of the West. Sigmund Freud extended his deeply pessimistic view of the individual into a generalised cultural pessimism. Modern art across Europe, from Dada in Berlin to surrealism in Paris, from T S Eliot's The Waste Land to Franz Kafka's The Trial, often reflected a relentlessly bleak vision. For the German sociologist Theodor Adorno, history did indeed have a direction, but it was a deeply depressing one. "No universal history leads from savagery to humanitarianism," he wrote in 1966, "but there is one leading from the slingshot to the megaton bomb".
At the end of the 1980s, the American historian Francis Fukuyama argued that this kind of pessimism had to be rethought. Though the first half of the 20th century had witnessed violence, death and destruction on an unprecedented scale, we should see this, in the larger scheme of things, as a historical blip. The world was now resuming its ineluctable path towards liberal democracy and freedom.
But Fukuyama was swimming against the tide. His optimism did not sit comfortably with the narratives of decline of the postmodern world. From the 1960s, pessimism in the west seeped into many areas of human experience as never before.
The first source of pessimism was the environment. After several million years of human history, the world's population reached two and a half billion in 1950. It took just 37 years for that figure to double. With the possible exception of some species of rodent, humans were now by far the most numerous mammals on earth, placing intense strain on the environment. Most experts predicted a global population of between nine and ten billion by 2050 - around four times that of 1950. As the biologist Edward O Wilson observed, we were "the first species in the history of life to go out of control on a global scale".
In the decade up to 1990, global industrial production exceeded that for all history up to 1950. The pollution it generated - in particular, the introduction into the environment of vast amounts of synthetic chemicals - created conditions for human and animal life that were outside the previous limits of biological experience. Never before had human activities impacted on the global climate and on conditions in the stratosphere. An environment that had taken millions of years to evolve was being significantly altered within the space of a generation. It was forcing human beings to make biological adaptations at an unprecedented speed.
The environmental resources of the planet could be seen as a form of "capital", which required stewardship on behalf of future generations. This "capital" had become dangerously depleted, particularly by the rich countries of the world. What generated the greatest sense of pessimism was our seeming incapacity to do anything about it. Many environmentalists already believed it was too late. As James Lovelock, the author of the influential Gaia thesis, put it: "It's like you're on a steep hill in a car and your brakes fail. You've got to do something and the least you can do is take your foot off the accelerator. But you're going to crash anyway."
A second source of pessimism was the idea of moral decline. Many saw the nuclear terror of the cold war, with its doctrine of mutually assured destruction (MAD), as a source of widespread moral contamination. Yet when the cold war ended, the "new world order" turned out to be just as morally repugnant. The gap between rich and poor countries steadily increased. With what the historian Eric Hobsbawm has called "the democratisation of the means of destruction", the affluent countries of the west could no longer consider themselves immune from the carnage and insecurity observable in other parts of the world. Long before planes crashed into the World Trade Center, military historians predicted the growth of new kinds of "informal" and "internal" warfare, presaging more repressive and authoritarian forms of government.
And torture, genocide and political murder persisted around the world - to a remarkable degree when you think that the evidence, in words and images, was more widely disseminated than at any other time in history. In the 1990s, one-third of the world's governments were said to be either using or tolerating torture. Since 1945, there had been 48 episodes of genocide, the majority taking place after 1965, with the loss of anything between nine and 20 million lives. Millions more continued to be in danger: one estimate in 1995 put it at 52 groups of potential victims in 36 countries - a total of 190 million people.
The west itself faced mounting evidence of a crime explosion and growing social disintegration. "Moral panic" - the dismissive term initially used by sociologists - gave way to "moral realism", as most criminologists accepted that at least some forms of crime were at historically unprecedented levels. Moreover, they had a disproportionate impact on those who were already economically and socially disadvantaged.
Explanations for the crime explosion focused either on the increasingly Darwinian qualities of late capitalism, or on the disintegration of the family and, with it, a secure environment for child-rearing. While neither explanation denied the possibility of moral choice, both placed an emphasis on structural conditions. At the end of the 1990s, these conditions appeared well entrenched.
In the academy, a kind of intellectual pessimism flourished, more arcane, but no less virulent. Though science had produced extraordinary material and intellectual benefits, many argued that the price had been too high. Science, its critics argued, possessed no inherent values, other than a commitment to its own methodologies and the knowledge it produced. This separation of knowledge from value made science too susceptible to the interests of those who funded it - not only the military-industrial complex, but also, for example, the genetic-industrial and the medical-industrial.
The critics of science argued that it was increasingly driven by the imperatives of profit, and they coined the phrase "dirty science". They also argued that science was wholly intolerant of other ways of knowing, and that it had depersonalised the world, prescribing a universe utterly devoid of human meaning or purpose.
It could be said that there was no point in looking for meaning in science, because that was to confuse science with religion. But in an increasingly secular world, religion did not have much to offer in the way of meaning either. So what about art - repeatedly called upon since the 18th century to fill the spiritual void bequeathed by science and the Enlightenment?
Many people still believed that this was art's highest function. But when they looked around them, they saw an art world devoid of moral authority, dominated by the logic of consumption. The high seriousness of modernism had degenerated into a weary and cynical postmodernism. The arts were now synonymous with the entertainment industry; judgements of the market were the only judgements of value. In such a culture, the idea of an art that might be ennobling or spiritualising was just not credible.
A fourth strand of pessimism arose from political incapacity in the face of the "new" capitalism. The "new" capitalism differed from the "old" (usually associated with "Fordism") in that it displayed more aggressive tendencies. It was certainly more "productive", but the price paid in the west was growing insecurity (often described as "flexibility"), higher unemployment, increased inequality and widespread social disintegration, with soaring rates of crime, divorce and family breakdown.
Investors in the money, bond and stock markets became the new policy-makers, as they acquired unprecedented influence over the economic life of nation states. Governments had always been subject to forces beyond their control; what was new was the sheer unpredictability of these forces and the speed with which unpredictable events would unfold.
The "new" capitalism first took root in Britain and the US. But its expansionist tendencies brought it into conflict with those capitalisms that reflected less individualistic traditions. The social market economies of France, Germany and Scandinavia, with their generous welfare provisions, were all threatened by the more ruthless regimes of capital accumulation; so was Japan, with its "planned inefficiency", a sort of privately funded welfare state, which kept unemployment low and maintained social cohesion.
The logic of the "new" capitalism seemed to demand its continued geopolitical advance. But where would the drive for competitive advantage end? Even if first-world consumption levels could somehow be extended to everywhere else, we were left with the prospect of the exhaustion of the planet's natural resources and a fast track to environmental collapse. As Hobsbawm argued, this latest phase of capitalism had reached a point of historic crisis, where the very structure of human societies was threatened.
So the pessimism unleashed by the current international crisis is part of a more general pessimism. It intensified and proliferated during the last decades of the 20th century and is now part of the zeitgeist. Environmental, moral, intellectual, political - it all coalesces to produce a pessimism that is truly cultural. But why don't we all share it?
First, pessimism is intimately bound up with values and ideology. It has no objective reality. One person's pessimism is another's optimism. For each person who despairs of the "new" capitalism, there is another who sees it as the engine of progress; for those pessimistic about our relationship to nature, there are others who celebrate the triumph of science. Even where values are shared, we will find that the same conditions produce both optimism and pessimism. This is partly a question of intellectual judgement; but this is not the only factor, or even the most important one. Cultural pessimism also arises from psychological disposition. Its defining characteristic is an attachment to negativity.
Pessimists share this disposition with those who are mildly depressed. Negative feelings and negative thoughts become intertwined, producing what psychologists call "a negative cognitive shift". Sometimes, pessimism can act as a bulwark against depression, a projection on to the external world of a negativity that would otherwise be directed towards the self.
Most experts agree that depression has been on the increase in many countries during the second half of the 20th century. The World Health Organisation expects it to be the number one health problem by 2020.
There are many studies that link this increase to the trends I have discussed: growing insecurity; feelings of inadequacy in a relentlessly competitive world; increasing inequality; and a powerful sense of social instability. If cultural pessimism is also linked to this, we can say that it exists in a double sense: it is both about our culture and produced by our culture.
Does this mean that pessimism distorts our understanding of reality? Is its attachment to the negative a kind of cognitive contamination? It would be reassuring to think so but, perhaps surprisingly, the evidence points in the other direction. It is not negativity that is most likely to lead us into error, but our deep need to comfort ourselves with positive illusions.
Numerous studies have shown that most people maintain unrealistically positive views. They tend to overestimate their ability to control their environment and to harbour unrealistic expectations of what the future will bring. Positive illusions of this kind are an essential part of maintaining psychological well-being.
We should therefore be cautious of dismissing pessimism as no more than the product of a temperamental defect. Just because it has featured so often in our intellectual history, it doesn't mean we should ignore it in the present.
A strong dose of depressive realism may be what we most need at present.
Oliver Bennett is senior lecturer in cultural policy studies at Warwick University and the author of Cultural Pessimism: narratives of decline in the postmodern world (Edinburgh University Press, £14.95)