Richard Holbrooke was very aware of the dangers of over-intervention. He had served in Vietnam and thought it was a fiasco. In Bob Woodward's book Obama's Wars, he appeared as a critic of the Afghan surge. But as the head of Afghan strategy, he was not able to stop it before his death last year. During his tenure, the number of international troops in Afghanistan was increased to more than 150,000 and the US was soon spending over a $100bn a year.
Even Holbrooke's relentless, bulldozing energy could not convince the US government that local Afghan politics and social structures would prevent Nato from defeating the Taliban or building a central state. His failure reflects the particular, 21st-century qualities of US and British foreign policy: an optimism, abstraction and isolation from local reality. These qualities were major factors in driving the west deeper into humiliating messes in Afghanistan and Iraq, with great sacrifice of human life and at a great cost to our credibility and reputation. That is why, although I am in favour of having a UN-backed no-fly zone in Libya to protect civilians, I am anxious about what happens next.
Much of the problem with foreign policy institutions such as the UN agencies, foreign offices, armies and development agencies is recent and stems from new obsessions, rooted in a new corporate culture. Officials in Kabul, whether they are diplomats in an embassy, UN staff in a compound or captains in a forward operating base, are on short tours, often six months long, rarely longer than two years.
Foreign civilians are trapped under security restrictions and curfews, in guarded embassy compounds, which keep them brutally isolated from the realities of local life. Too few senior officials involved in Afghanistan speak Afghan languages well or have served repeated postings there – and that is after we have been heavily involved for more than nine years.
Such problems are not unique to Afghanistan. Too few British ambassadors in the Middle East speak fluent Arabic. The core competencies that govern promotion in the Foreign Office are based on abstract management and policy skills, rather than deep knowledge of a country. A friend of mine was recently told in an interview for a diplomatic post that he was not allowed to talk about his previous postings in the region for fear that it would prejudice the process. Much of this is the result of deliberate management changes, introduced over the past ten years.
But the Foreign Office is still in a better position than most of its bilateral and multilateral peers. The US state department's 2010 quadrennial review calls for even more emphasis on importing staff from domestic departments and more emphasis not on traditional diplomatic skills but on global skill sets. More and more people will be promoted for moving fluidly through the abstractions of global and multilateral issues. In Kabul, the leading international development agencies and institutions all employ far more international consultants on short contracts than core staff. Staff switch between sectors or even government departments. They are rarely given the time or the incentives to specialise in the place or the language. Their colleagues and bosses, who are also non-specialists, change continually and move swiftly on before their proposals can be tested, still less fail. A culture of country experts has been replaced by a culture of consultants. This new culture can be beneficial for management practice and multilateral diplomacy. Modern ambassadors are better at managing their budgets and staff than their predecessors. Contemporary diplomats operate well with other diplomats at the UN and in policy discussions on climate change, trade and other global issues. But a more globalised world also requires particular knowledge of power politics in specific countries. And this is true of our most extravagant and dangerous international exercises: interventions.
Lack of knowledge of specific countries may make us more, not less, likely to over-intervene. In Victorian Britain, the most jingoistic imperialists were often those who never left Britain: those with the deepest contact with ground reality were often the most cautious. This is one of the reasons why, although the British got stuck in humiliating messes in Afghanistan in 1842 and 1880, they recognised their predicament more rapidly and withdrew their troops. The same was true in Iraq in 1920.
This was not because of an absence of public pressure. Many Victorian public figures in London made abstract arguments that we had an obligation to remain in Afghanistan; that the Afghans wanted us; that, if we left, it would become a failed state, exploited by Russia, endangering the whole region. If we were seen to be driven out, British credibility would be destroyed around the world. But it was General Frederick Roberts, who had been on the ground commanding in the second Anglo-Afghan war, who was able to reject these views and say in 1880 (in a way that General David Petraeus would not): "We have nothing to fear from Afghanistan and the best thing to do is to leave it as much as possible to itself. It may not be very flattering to our amour propre but I feel sure I am right when I say that the less the Afghans see of us, the less they will dislike us."
Don't misunderstand me. Victorians were often racist, brutal and exploitative. There is a limit to comparisons across time. We neither could nor should ever want to re-create Victorian structures, but there is something relevant in their professionalism and knowledge.
Roberts came from a system whose career structure was designed to reward long experience, those who had served in remote posts and displayed detailed knowledge of specific cultures. That experience is partly why Victorian soldier-journalists such as Archibald Forbes, or administrator-politicians such as John Lawrence, or generals such as Roberts, whom you might expect to defend the establishment line, were often more critical than their equivalents today. Their immersion in other cultures seems to have made them more confident, more cautious, more concrete and more critical.
In 1892, Forbes wrote about the murder of civilian British envoys whose corpses were paraded through the streets of Kabul in 1841. Alexander Burnes and William Hay Macnaghten had met their fate because they had gone to Kabul as "supporters of a detested intruder and the unwelcome representatives of a hated power". As late as the 1920s, Gertrude Bell, serving as senior political officer in the British occupation of Iraq, wrote a series of semi-private letters to ministers and journalists. They were never muffled in platitudes: "There's no getting out of the conclusion that we have made an immense failure here. [In talking to an Arab nationalist leader] I said complete independence was what we ultimately wished to give. 'My lady,' he answered (we were speaking Arabic), 'complete independence is never given; it is always taken.'"
By contrast, today's rapidly improvised and ad hoc collection of internationals in Kabul and Baghdad, with their extravagant budgets, ill-defined powers and very short-term postings, can be more entranced by rickety ideas and dangerous projects. The lack of deep immersion or specialisation often results in heady optimism and abstraction.
The first was epitomised in the Afghan intervention by the US military. Commander after commander arrived, declared that he had found a dismal situation, announced a new strategy requiring new resources, proclaimed a decisive year and then left a year later, optimistic. Then, a successor arrived, identified a dismal situation, reversed the previous policy (going from centralisation to decentralisation, mentoring to partnering, or vice versa) and proclaimed their own decisive year. And so on.
Abstraction, meanwhile, was personified by the civilian policy elite, who often spoke about Afghanistan – a country of 20,000 diverse, impoverished, traumatised villages, each with its own distinct and often successful forms of organisation – as though it were a blank space on the map. On to this, they imposed phrases such as "the rule of law", "governance", "the legitimate monopoly of the use of violence" and "chains of accountability linking the citizen to the state".
Such vagueness and imprecise phrases were heaped in countless combinations in strategic plans and mission statements to justify countless endeavours. Inevitably, policymakers succeeded best in those things that could be done from offices in the capital but performed increasingly badly the closer they came to the real structures and forms of resistance in Afghan rural life. Thus, for example, they did well at stabilising the currency but very poorly at establishing honest local policemen; well at designing bridges, poorly at weaning farmers from opium poppy production.
The same abstraction of language allowed think tanks in Washington and London to conjure nightmares, which were so indistinct and ghostly that they could neither be defined nor refuted – and were all the more terrifying as a result. Of these, the greatest were the fears of failed states, regional instability and global Islamic extremism. As this paranoia became ever more isolated from reality and ever more intense, it made failure (which was inevitable) paradoxically inconceivable. Failure was so mysterious and horrifying that it became not an option. As their goals were so abstract and so difficult to relate to what happens in an Afghan village, failure also became invisible. (How would you know whether or not you were creating Afghan governance?)
Policymakers neither wanted to nor could acknowledge that their policy was failing. And, because it had clearly not yet succeeded (what, after all, would success look like?), they sent in more money and more troops and made more plans. Holbrooke thus faced institutions in which isolation, optimism, abstraction and a lack of country knowledge led inexorably to over-intervention.
What are the policy implications of all of this? The first is that, given the frailties of 21st-century institutions, we should be far more wary of intervention today than at any time in our history. Full military intervention now is like an amateur attempting high-altitude mountain rescue. It can be heroic but the chances are against it doing much good, for either the rescuer or the victim. We need to acknowledge how ill-equipped we are, how little we know about these particular mountains.
And we need to be aware of our tendency to ignore the dangerous weather and, through some abstract optimism and confused sense of obligation, climb further into failure and tragedy. What advice could you give such a climber or intervener? Prepare as well as you can; avoid the most difficult routes and do only what is within your capability. There are certain dangers, certain types of difficult terrain that you can avoid. Other things, such as a blizzard, cannot be avoided; they can only be monitored. And if conditions become impossible, however much you have sacrificed or invested in the situation, turn back. The most useful thing that we can do in the long term is to build a corps of more experienced guides for foreign policy. Much of their work will be involved with commerce and with multilateral and human rights issues; only some of it with more extreme measures, such as sanctions and no-fly zones.
Their expertise should certainly not make us intervene more or entice us with a promise of better occupations. Frequently, their role will be to make the detailed, country-specific arguments about why we cannot intervene in a particular place or why we should not intervene too deeply (as we were sucked too deep into Afghanistan after 2005).
Their role would be not just in limiting our initial interventions but in reassessing them; spotting the dangers; acknowledging the failures; and having the confidence, if necessary, to pull back. And, in cases such as Bosnia, where intervention was the right policy, giving us the confidence to continue.
At the moment, in Libya, we can resist putting boots on the ground because of the language of the UN resolution. But we will soon need much more than abstract legal theory. If Colonel Gaddafi were still in place in three months' time, or there was a humanitarian catastrophe despite our no-fly zone, or the opposition were crying for help, how would we respond? And how would we respond to a civil war in which both sides killed civilians? Or to a post-Gaddafi state in which former regime elements mounted an insurgency?
These are scenarios that are best imagined and understood by people who know the place well and have known it over many years. We need people who understand the Italian colonial history, the past ten years in Benghazi and the sorts of networks Gaddafi still possesses. They would need to be able to judge whether the power of tribal groups is fading and how much emphasis to put on Islamist parties and how far to trust them. And they would need to be able to do this in situations of stress and chaotic unpredictability.
The confusion and uncertainty in North Africa and the Middle East today are thus an urgent reminder of our need for people who have a profound understanding and kinship for other cultures. If this is compatible with skills at internal management and the jargon of multilateral global issues, so much the better. But it is the practical understanding of the country and the relationships with its people that are the priority. William Hague, in the Foreign Office, seems aware of the problem and has launched an initiative on diplomacy. But we are yet to see any bold reforms. If we are to undertake more interventions, then our international institutions, which have for so long lectured developing countries on governance and civil service reform, will need to be healed themselves. And they may have to turn away from their obsession with contemporary management best practice and look at less comfortable models.
In the 19th century, Charles Trevelyan and his predecessors completely recast the civil service through the introduction of new entrance examinations, professional schools and legal requirements that the majority of those controlling policy in London had served a posting in the country for which they were responsible, and laws ensuring that the most senior posts in the country went to people with long experience on the ground. They did so to control the demons of the 18th century: amateurism, nepotism and corruption.
Our reforms should be as ambitious and should be targeted at the demons of the 21st century: isolation, optimism and abstraction. These reforms must happen not just in Britain but in the US and the UN. If we reform our institutions, we can responsibly exercise noble options such as humanitarian intervention. But if not, these adventures will become only more ill-informed and more dangerous. Failure will become increasingly inevitable, invisible and inconceivable. We will discredit the interventions and ourselves.
Rory Stewart is the author of "The Places in Between" (Picador, £8.99). In May 2010, he was elected Conservative MP for Penrith and the Border