It sounds crazy, given how often we refer to it in everyday life, but the idea of luck has been under attack.

Denying the importance of good fortune is central to the way we like to talk about success. In a meritocracy, the winners deserve to win, or at least they think they do. That was one of Michael Young's central objections to the concept when he coined the word in his satirical book The Rise of the Meritocracy, first published in 1958. Young predicted that vast social problems would follow the central logic of meritocracy, which he described as "talent plus effort equals merit". But how would life's losers feel about justified failure? And how would the winners behave, as they celebrated their justified success?

Later, writing for the Guardian four years in to Tony Blair's government, Young felt that history had proved him right:

. . . the rich and the powerful have been doing mighty well for themselves . . . If meritocrats believe, as more and more of them are encouraged to, that their advancement comes from their own merits, they can feel they deserve whatever they can get.

They can be insufferably smug . . . The newcomers can actually believe they have morality on their side.

So assured have the elite become that there is almost no block on the rewards they arrogate to themselves. The old restraints of the business world have been lifted and, as the book also predicted, all manner of new ways for people to feather their own nests have been invented and exploited.

Salaries and fees have shot up. Generous share option schemes have proliferated. Top bonuses and golden handshakes have multiplied.

Those perceptive paragraphs have even more power a decade later, in the aftermath of the financial crisis and the scandal over MPs' expenses. For all its lip-service to meritocracy, this generation's officer class has a very strong sense of entitlement.

The irony, of course, is that real meritocracy - which implies a decreasing correlation between privilege and achievement - appears to have gone into reverse. Astonishingly, that decline applies even to the supposedly open world of professional sport. On the 1987/88 cricket tour to Pakistan, England selected only one privately educated player. In contrast, about two-thirds of today's England team attended a private school. That means that 7 per cent of the population is supplying two-thirds of the cricket elite. In probabilistic terms, being privately educated (as I was) makes it about 20 times more likely that you will play for England (as I did). The same trend applies to England rugby players and British Olympians.

Sport, in this instance, reflects society. In 2009 the former health secretary Alan Milburn chaired a government-appointed panel to investigate social mobility in the UK. "Birth not worth has become more key to life chances," Milburn previously said. Social luck, the philos­opher John Rawls's phrase for privilege, shows no sign of loosening its grip.

Worst of all, our fortunocracy insists on a meritocratic pretence. Mockney accents, dumb­ing down, pressing meritocratic buttons, covering your privileged tracks: it's all passed off as modesty, as the Everyman touch. In fact, it is quite the opposite. Pretending to have fewer advantages than you did is not only a form of deceit, it is also form of conceit - an attempt to deny your good luck in order to claim more credit for yourself.

This discrepancy between myth and reality is equally relevant to those who are less fortunate. The language of self-determination has increased - social mobility, self-help, making your own luck. But that aspiration is not reflected in reality. So we are obstructed, in language and in thought, from bridging the gap between the mantras of modern society and self-help and the realities of modern life. A most uncomfortable question follows. Must there not be a point at which the philosophy of self-help and positive thinking borders on cruelty? Might it not be that believing in "making your own luck" is serving only to make us unhappy and confused?

There is a second, separate strand to anti-luck. Those who deny the existence of innate talent also set out their stall against luck. People whom we once called geniuses must now be rebranded as normal folk who dragged themselves towards brilliance through work and an effort of will.

Blessed freaks

Malcolm Gladwell's book Outliers, published in 2008, popularised the idea that practice and intensely hard work, not talent, ultimately determine success. Matthew Syed's Bounce proposed that geniuses - from Mozart to Roger Federer - are "made" rather than "born". And Daniel Coyle's The Talent Code argued that talent was a myth, a hoax, an old wives' tale; even worse than that, talent was a conspiracy.

Syed, a Times colleague whom I admire even when I do not agree with him, put it like this: "It is Federer's regular practice that has given him his advantage, not his genes . . . Top performers are not born with sharper instincts; instead, they possess enhanced awareness and anticipation . . . The key thing to note is that these cannot possibly be innate skills [my italics] . . ."

Why not? Surely it is inevitable that some people have a genetic predisposition to become great athletes, and that this talent must then be honed by many thousands of hours of practice. Almost everyone who has played sport, amateur or professional, knows that "natural sportsmen" exist. As Rafael Nadal said of Federer (and Nadal would know): "His physique - his DNA - seems perfectly adapted to tennis. You get these blessed freaks of nature in other sports, too."

How perverse that agreeing with Nadal has become almost taboo, that talent has been turned into a topic of reproach. One of the central pleasures of watching sport, or taking in the performing arts, is recognising God-given talent (or luck-given talent, for the secular-minded), and not only admiring guts or determination. Often, it is when pure talent takes over, producing a performance that no coach or training regime could orchestrate, that sport makes the soul sing. Writing genetic good fortune out of the script does not enhance life. It obscures the obvious truth that true greatness will always depend on a mysterious compound of genetic blessings and hard work.

How did I become obsessed by luck? As a professional sportsman, I was conditioned to deny its existence, telling myself that my destiny was always in my own hands. But I learned the hard way. In 2008, when I was captain of Middlesex, my team was on the brink of its fifth consecutive win when inexplicably I broke my ankle during a routine passage of play. I never played professional cricket again. The silver lining of that moment of bad fortune was that it taught me about luck in a much broader sense.

I might never have examined properly how my education had stacked the odds in my favour if I hadn't first cursed the bad luck of having my ankle in a cast.

While I was hobbling around Lord's, a thoughtful Middlesex fan gave me a copy of Fooled by Randomness, the book that made the trader-turned-author Nassim Nicholas Taleb into the world's best-known scourge of the banking system. The previous summer I had been asked to review Taleb's subsequent book, The Black Swan (2007), but I'd been flat out captaining Middlesex and was obliged to decline. Now, a year on and with my ankle in plaster, I had more than enough time on my hands. You could say it took my black swan to find Taleb's.

I arranged to meet up with Taleb, and the day we chose, weeks in advance, was 10 October 2008. That turned out to be Black Friday. As the banking system tottered on the brink of collapse, Taleb's stock rose. I was interviewing a theorist of chance and randomness on the very day, randomly selected, that he described, only partly exaggerating, as "the most important day of his life".

Over the following 18 months, as I researched my own, very different inquiry into luck, I found myself increasingly drawn to the financial crisis. First, because it was - is - the story of our times. Second, because many of the books that already existed about luck are dominated by finance. If you try to understand luck, you can't help but follow the money.

Event horizon

Luck is not something to which financiers like to admit, any more than sportsmen do.

Indeed, one of the finest financial books ever written, Against the Gods by Peter L Bernstein, casts luck as the villain of the story, the bad guy on the wrong side of history. Luck, according to him, is gradually being supplanted by a better, more scientific understanding of the world. Luck is the gloomy fog slowly being burned away by the light of human progress, embodied by the concept of risk.

“The revolutionary idea that defines the boundary between modern times and the past is the mastery of risk: the notion that the future is more than a whim of the gods and that men are not passive before nature. Until human beings discovered a way across that boundary, the future was the murky domain of oracles and soothsayers who held a monopoly over knowledge of future events."

The useful concept of risk, according to Bern­stein, has helped to free society from the shackles of fate and superstition. His heroes range from Renaissance gamblers to the inventors of modern financial models. Their urge to predict, to calculate and to quantify has beaten back the forces of rogue chance.

One of the greatest of Bernstein's heroes is still alive - Kenneth J Arrow, professor emeritus at Stanford University and joint winner of the 1972 Nobel Prize in Economics. In the spring of 2009, I visited Arrow at Stanford to ask him about the ascent of the idea of risk (or descent, given the financial turmoil at the time). Though he was 87 years old, he wanted to meet in the hurly-burly of the campus café. He is still famous, and undergraduates edged towards him, hoping to catch snippets of conversation from the great man as he ate his sushi. Small, neat, casually dressed in a beige shirt and above all unfailingly humble and polite, he is nothing like the big players in the testosterone-driven world of high finance.

Yet Arrow is one of the fathers of modern economic theory. In particular, his work opened the way for the explosion of derivatives - or "financial weapons of mass destruction", as Warren Buffett has called them. So, as we chatted about the financial crisis in the Californian sunshine, it felt a little like interviewing a disinterested scientist who had pioneered nuclear science only for him to watch in horror as it evolved into the atomic bomb.
After all, according to Arrow's work, derivatives ought, in theory, to have worked for the public good. As he wrote in a blog responding to the 2008 financial crisis: "In the 1950s, it was shown . . . that increasing the number and coverage of risk-bearing instruments [derivatives] would improve the running of the economy."

So, what happened? "The real question is why banks like Bear Stearns and Lehmans didn't see the risk," he said. "Surely these people were capable of evaluating the risk? Well, I was wrong. And I still don't understand what caused them to be so wrong. There was deregulation; that's part of the story. But you would have thought even without regulation, the self-interest - the greed, if you like, the cautionary greed - would have prevented this from happening."

In truth, as we now know, the idea that risk had mastered uncertainty, the Apollonian ideal envisaged in Against the Gods, brought with it new risks. Remember Matthew Rothman, the quantitative risk expert at Lehman Brothers when it collapsed? As he infamously said, reacting to the turmoil: "Events that models only predicted would happen once in 10,000 years happened every day for three days."

To some extent, the crash was a reprise of a familiar story. According to the financial wizards behind Long-Term Capital Management in the late 1990s, investing was no longer subject to chance at all: it was pure science. Mastery of probability and risk meant we were too clever to need to be lucky. "In a strict sense, there wasn't any risk," said Merton Miller, another Nobel economics prizewinner, of LTCM's celebrated Black-Scholes model, "if the world behaved as it did in the past."

This model was a mathematical formula, devised by two Nobel laureates, that traders believed they could use to hedge against any bet on share options. When the world unhelpfully failed to behave accordingly, LTCM went spectacularly bust. Even then, Miller conveniently wondered whether "the disaster was merely a unique and isolated event, a bad drawing from nature's urn". Quite the reverse: it was the idea that there couldn't be a bad drawing from nature's urn. The problem wasn't bad luck. It was the vainglorious refusal to believe that the idea of luck had any relevance.

Denying the existence of luck appeals to the basic human urge to control everything - a neurosis that affects almost every aspect of our lives. It is difficult to accept that we are all, to some degree, victims and beneficiaries of circumstance.

It may be difficult but it is surprisingly practical. Our understanding of evolution shows that success relies on the interaction of chance mutation and natural selection. The point here is that we cannot say the successful evolution of an organism is caused by 60 per cent chance mutation and 40 per cent selection. They do not "mix"; they interact to produce something quite new. Chance is a crucial ingredient that goes into making an end product that may be unrecognisable from its constituent parts.

I would make the same argument about an individual life. We are misled by the autobiographies of great men and women who claim, after the fact, to have planned their ascent meticulously, to have converged on success like soldiers finding a flag in an army training exercise.
The origins of success are usually much more subtle and complex. Successful people, by being open to opportunity and exposing themselves to chance, take new directions that prove more fruitful than anyone could have predicted. A life does not "follow a course". We change in many ways as we grow. A missed opportunity represents the failure to evolve into a different, better person.

Believing in luck does not imply fatalism, as many people mistakenly believe. But it does demand openness - and humility.

Ed Smith's "Luck: What It Means and Why It Matters" is published by Bloomsbury (£16.99)