Isaiah Berlin’s famous essay The Hedgehog and The Fox (1953) divided thinkers into two categories: foxes, generalists who know many things; and hedgehogs, who know only “one big thing”. Nearly 60 years later, it is clear that the specialist hedgehogs have had a very good run – certainly in the workplace. But to see just how far the foxes have been pushed towards extinction, consider three stories from the past five years: the financial crisis, MPs’ expenses and phone-hacking. In each case, the reaction was as revealing as the crisis itself. It was blithely assumed by many that the disasters were inevitably caused by a lack of expertise and “professionalism” – and that the solution is always ever more specialisation and compliance.

Too many experts

But what if the problems at the top of our most important institutions are not caused by an amateurish lack of expertise but quite the opposite – by professional insularity and a lack of human perspective? If so, the conventional res­ponse to each crisis (demanding that the next elite is more expert and has flawless credentials) may serve only to compound the underlying problem: the lack of good leadership.

Look again at the narrative of each crisis. During the early stages of the financial crisis, angry commentators scoured the boardrooms of failing banks looking to expose men and women who weren’t financial experts. The satirists John Bird and John Fortune mocked Matt Ridley, the geneticist and science writer who was chairman of Northern Rock when it failed, for his lack of grounding in financial services. Pillorying apparently underqualified people who had wandered out of their depth was a common response to the bank bailouts.

But today’s bankers are more specialised and technically knowledgeable than any in history. The proportion of maths and economics graduates in banking has risen as the number of history and arts graduates has dwindled. No generation of bankers has ever studied finance to such a relentless extent.

Finance is just one manifestation of the ­triumph of the technocrat over the generalist. But at what cost? It may require a specialist to understand a single complex derivative. But it takes a generalist to recognise that the array of opaque financial instruments traded within just one large modern bank cannot be understood by any single human mind, no matter how sophisticated or expert.

Three years ago, I asked John Kay, who chaired the review of UK equity markets and long-term decision-making, if he thought that the old City gents, for all their weaknesses, would have been better equipped than today’s hyper-specialists to avert the crisis. “Very clearly,” he replied. Following his own logic, Kay’s review on reforming the financial sector could go beyond recommending the separation of retail from investment banking and also suggest that bankers pursue broader intellectual interests. Niall Ferguson made a similar point in his recent Reith lectures. The first rule of financial regulation should be simple: bankers should learn more history.

The financial crisis is a parable about what happens when “expertise” holds the whip hand over integrity. Say what you like about Bob Diamond but he definitely understood banking – all too well. 

The reaction to the MPs’ expenses scandal also reveals the assumption that grave institutional crises are caused by a lack of “professionalism”. The underlying logic was that British politics needed to evolve into a bona fide profession with clearly defined rules and conventions. But it was British politics’ evolution into a dedicated profession that helped create the expenses scandal. Today’s MPs know more politics – and more often only politics – than any previous generation. They know where the “swing” votes are and many spend their whole professional lives studying how to attract those votes. Becoming a clique of professional experts, increasingly cut off from the outside world by their collective monofocus, was ­exactly what led them to abuse the expenses system. The scandal wasn’t about an unmodernised old-world club. It derived from an all too modern sense of professional entitlement.

What about phone-hacking? Consider the initial defence: a single rogue reporter, a few lazy hacks, over-reliance on unprofessional private investigators. In fact, the culture of phone-hacking did not evolve from people who didn’t understand the news. It emerged from the demands of people who understood the news too well and who grasped, with brutal clarity, what sells newspapers and how to exploit the business of titillation and speculation. Phone-hacking was a symptom of the professionalisation of the gossip industry – and the sense of detachment that went with it.

Rise of the hedgehogs

All these failures of leadership – in politics, in finance, in the media – followed not because people did not understand their own industry. The problem was that they understood only their own industry. The central cause was not insufficient expertise but a lack of judgement – a failure of judgement caused by a lack of perspective and balance.

In a famous experiment, Philip Tetlock, professor of organisational behaviour at the University of California-Berkeley, interviewed 284 experts who made their living commenting or offering advice on political and economic trends. He asked them to assess the probabilities that certain events would occur and then calculated how often the experts were proved right. The results were dismal, even if some types of thinkers fared better than others: “The better forecasters were like Berlin’s foxes: self-critical, eclectic thinkers who were willing to update their beliefs when faced with contrary evidence and were doubtful of grand schemes . . . The less successful forecasters were like hedgehogs.” 

Tetlock’s study focused on the rightness (or wrongness) of expert pundits. The much more important lesson revealed by the three moral crises spread over five painful years is not only that foxes make better pundits. They also make better practitioners. 

Ed Smith’s “Luck: What It Means and Why It Matters” is published by Bloomsbury (£16.99)