When I joined the Foreign Office in 1979 it was engaged in a soul-searching review into why it had failed to predict the fall of the shah of Iran. On the eve of the revolution, our ambassador in Iran, Anthony Parsons, had embarrassingly written to David Owen, then foreign secretary: "The shah remains in complete control of the country and of the government . . . I do not foresee any serious trouble in the near future."
More than 30 years later we failed again, this time to predict the Arab spring. And again there has been soul-searching. Meir Dagan, the former head of the Mossad, said defensively: "We didn't anticipate the timing and we didn't anticipate the magnitude, but we did think there were severe structural problems. It is important to say that, in terms of the intelligence agencies, their principal focus is not the people but what the governments think. If the governments are surprised, we too are going to be surprised."
But why should we be able to predict such events? The shah didn't know the revolution was coming, even though Savak, his secret police force, penetrated every corner of the country. In Egypt, Hosni Mubarak didn't know his downfall was imminent. Colonel Gaddafi did not recognise the serious challenge he was facing in Libya even after the uprising in Benghazi had begun. So why should we? Revolutions are intrinsically difficult to predict.
Just as we were wrong in predicting how the Iranian Revolution would play out back in the 1970s we will be wrong, in all probability, in our predictions of how the Arab spring will develop. In my view, we were too starry-eyed at the beginning of 2011, not envisaging the obvious problems that would arise, and we are too gloomy now, expecting the whole enterprise to collapse into chaos just because the voters in Egypt, Morocco and Tunisia have opted for Islamist parties in free and fair elections. The process will be difficult and will suffer frequent setbacks, but that the Arab peoples have begun to throw off long-established tyrannies is unalloyed good news for us and for them. They deserve our support, not least to avoid making the mistakes that bedevilled the countries of central and eastern Europe when they threw off the yoke of communist dictatorship.
The Arab spring started when Mohamed Bouazizi, a street vendor in a town in central Tunisia, was personally humiliated by an official in December 2010 and subsequently burned himself to death. That single, desperate act inspired uprisings across the region. Many Arab governments were rotten and hollowed out, held up only by hubris. Experts told us that we didn't understand, that the Egyptian people loved Mubarak and that the army would never let him be brought down. Or that the Libyan people, apart from those in Benghazi and the east of the country, loved Gaddafi. Yet the people hated the dictators, and particularly their attempts to make their reigns hereditary. No new blood was coming into the system to revivify these countries. There were no democratic elections to let people change their governments peacefully and no freedom of expression to allow people to let off steam. The choice was between no change at all and revolution - and the people chose revolution.
It is important to understand, however, that the revolutions were not just about democracy and fundamental rights. The causes were multiple and complex. People were equally fed up with the corruption and incompetence of the regimes and with their inability to provide jobs or food.
It is still uncertain how far the revolutions will spread. In the 1990s, Algeria endured a decade of searing violence in a civil war between Islamist extremists and a vicious military government. The public is not sure whether it wants to risk such suffering again by opting for a revolution, and yet it is hard to believe that the rickety regime will remain in place for long. The kings of Morocco and Jordan have tried to get ahead of the revolutionary spirit by introducing tentative democratic reforms but it remains to be seen whether these will succeed in taking the heat out of the revolutionary zeal.
In the Gulf states, with the exception of Yemen and Bahrain, there have been only minor skirmishes so far. In Yemen, the effort to remove President Ali Abdullah Saleh has been complicated by intertribal rivalries, the desire of many in the south to separate once again into two countries, and a battle in the north between the Shia Houthis and Salafist Sunnis. In Bahrain, there is an unstable stand-off between a Shia majority and a ruling Sunni minority.
The most important remaining front is Syria. I asked a leading official in the region how soon Bashar al-Assad would fall. He thought for a moment and replied: "Somewhere between one week and three years." A rather wide time frame, but it is clear that the ruling clan will have to go, given that people on the streets have broken through the fear barrier.
No one knows when the Assad regime will fall but, as in Libya, it will reach the tipping point at some stage in the coming months and go quite suddenly. After it has gone, Iran will be next, in my view. The Iranians have been helping the Syrians to sharpen their techniques of suppression. Once young Iranians see that these methods do not work and that the corrupt Assad regime can be overthrown, they will feel emboldened to take up their unfinished revolution once again.
The uncomfortable question that Syria raises for the west is whether or not to intervene. After the war in Iraq, the commentariat announced that liberal interventionism was over; no one would ever make that mistake again. Yet only a few years later the west has intervened once again, successfully, in Libya in support of the rebels. David Cameron and Nicolas Sarkozy did not secure a UN resolution in favour of regime change any more than the Blair and Clinton administrations did in Kosovo, but Nato's military intervention in Libya did help bring about a change of regime. The Prime Minister deserves considerable credit for going out on a limb.
Should the west do the same thing in Syria? Can we stand by while Assad's regime murders the Syrian people? Can we risk the civil war that the regime, in its frantic attempts to survive, is trying to provoke between the minority Alawite sect, to which most of its leading members belong, and the Sunni majority? If such a civil war spreads across the whole arc of the Middle East where Sunni and Shia communities rub up against each other, from Lebanon and Iraq to Iran, Saudi Arabia and Bahrain, the consequences for all of us will be profound.
The best criteria for reaching a judgement on intervention are the five tests set out in Tony Blair's 1999 Chicago speech, made in the context of the Kosovo conflict:
- Are we sure of our case?
- Have we exhausted all diplomatic options?
- Are there military options that we can sensibly and prudently undertake?
- Are we prepared for the long term?
- And do we have national interests involved?
In Syria, many more opposition supporters and innocent bystanders have been killed than in Libya and, thanks to the opposition's skilful use of technology, the pictures of the outrages are on our television screens every evening.
So the case for our intervention to protect civilians is every bit as strong as in Libya, but our action would be viable only if the rebels wanted us to intervene. So far they are divided on the matter, with some calling for international military support and others opposing it. In any case, military action would be practicable only with the support of Syria's neighbours. Turkey, Saudi Arabia and Jordan are calling on Assad to go, but they are still ambivalent about military action. There remains a significant segment of the population loyal to Assad, partly for fear of something worse, and military intervention would be difficult and bloody.
Most of all, we would be wise, after the experience of Iraq and Afghanistan, not to go in until we had a clear plan for the aftermath and were convinced our publics had the patience to endure long-term involvement and the concomitant casualties. Though now is not the moment to contemplate military intervention in Syria, we would be foolish to rule it out. If civil war on sectarian lines does take off we will not be able to stand on the sidelines indefinitely, because the conflict will not remain confined to Syria.
Even if we do not intervene militarily, we need to start thinking about the political situation in a post-Assad Syria. In Iraq, we made the mistake both of leaving it too long before starting a proper dialogue between the Sunni minority and the Shia majority and of pressing ahead too rapidly with de-Ba'athification, just as we did with de-Nazification in Germany after the Second World War. In Libya, we took a risk by backing the National Transitional Council and making no effort to organise a wider dialogue so that all forces could feel included, and we are now paying the price in sporadic fighting with the Touaregs and various armed militias. In Syria, the challenge is even greater because of the differences between the internal and external oppositions, and because of sectarian and ethnic divisions between Sunnis, Alawites, Christians, Ismailis, Druze and Kurds. This time we do need to ensure that such a dialogue starts early and includes those elements within the power elite which will be necessary for continuity once the small circle around the Assads leaves. We will also need to be prepared for the further radicalisation of Hezbollah, which will feel cut off in Lebanon by the collapse of its crucial ally.
The west has been accused, understandably, of double standards in the Middle East, of turning a blind eye to governments that did not share our values, because they were strategic allies or because of the dependence of our societies on the hydrocarbons their countries produced.
We have been criticised - wrongly, in my view - for double standards in opening relations with Gaddafi when he abandoned his chemical weapons as well as the ambition to develop nuclear weapons. In that case, reaching out to a reformed sinner was the right thing to do. It is crucial to offer incentives to regimes that abandon aggressive intentions and capabilities, even if we do not approve of their lack of democracy or human rights. In the words of a Yugoslavian proverb favoured by Franklin D Roosevelt, "When a river you reach and the devil you meet, with the devil do not quarrel until the bridge you cross." The one proviso is that, in the act of welcoming a reformed sinner, we should not make it more difficult for their citizens
to rise up against them. That Gaddafi fell to a popular revolution just seven years after we reopened relations demonstrates that the west did not entrench him in power by welcoming him back into the international community after he disavowed weapons of mass destruction.
Even if there is some truth to the accusation of western double standards in the Arab world, we cannot be neutral about what is happening in Syria and other parts of the region. The Labour Party, a party of the centre left, of internationalism, the party that supported the republicans in the Spanish civil war, should be absolutely clear where its heart lies.
We cannot be ambivalent about whether or not we favour democracy. If we are in favour of free elections then we don't get to choose who wins them - the Arab people do. We must accept the outcome of those elections even if, as in Egypt, that leads to the election of the Muslim Brotherhood and the Salafists as the two biggest parties. The west made a grave mistake in 2006 when it rejected the outcome of the Palestinian elections because the people voted for Hamas. We should not repeat that mistake.
Some right-wing commentators, particularly in Israel, are suggesting otherwise. Dr Guy Bechor of the Lauder School of Government at the Interdisciplinary Centre, Herzliya, wrote recently that Barack Obama "will be remembered for ever . . . for destroying stability and national leadership in the Arab world. Jimmy Carter did that in Iran, and Obama has completed the process in the rest of the region" as the US topples its allies. For Bechor, democracy "is the shortest path to sharia", and he argues that the US, in opening Pandora's box to steal a glance at democracy, has released all the demons that will destroy its old friends as the Arab world returns to the Middle Ages.
It is natural that many people in Israel should be disoriented by the sudden changes going on all around them and uncertain about what the future holds. But the cult of stability is overdone. I lost count of the times colleagues told me when I was a diplomat that we should hang on to familiar leaders for fear of something worse, the most ludicrous example being "he's the best Brezhnev we've got". Evil regimes are a threat to, not defenders of, long-term stability and prosperity - and they are best overturned. Israel was never going to be able to depend on the acquiescence of dictators for its stability. It is a curious notion that the spread of democracy in the Arab world could undermine a democracy such as Israel. Indeed, history tells us that democracies are far less likely to pose a threat to their neighbours than dictatorships. Removing the dichotomy that separates views on the Arab street from the private views of their rulers should be a good thing for Israel and should yield more honest diplomacy.
Interestingly, recent opinion polls show that the people in the newly free Arab countries, like the people of Israel, overwhelmingly see Iran as the biggest threat. The revolutions in the Arab world may even make progress on a Middle East peace process more likely by "shaking the kaleidoscope" in the region and breaking up the stasis that has prevented credible negotiations for so long. Certainly the more proactive and more positive role played by the Arab League in the case of Syria, something previously unheard of, may be a harbinger of good things to come.
That is not to say that elections alone will solve the huge challenges that Egypt and its neighbours face. The revolutions were not just about freedoms and about voting, but also about corrupt and incompetent governments that could not provide for their people. If the new democratically elected governments cannot deliver better lives for their people they will be thrown out at subsequent elections. This could result in the sort of instability we saw in some central and eastern European countries, where a succession of weak and incompetent governments rolled out in the aftermath of communism. It is very much in our interest to support new democratic governments in the Arab world to ensure that the same does not happen, both with a programme of aid of the sort that the Marshall Plan provided for western Europe after the end of the Second World War and even more importantly with training and expertise. We should offer help in applying the rule of law to prevent corruption and provide strong, fair courts to administer justice equally. We should also support government structures that can deliver in terms of food and jobs for the burgeoning young population. This challenge is above all one for the European Union: we cannot expect the US to lead in our neighbourhood on recovery and rebuilding any more than it did in the war in Libya.
One outcome of the Arab revolutions has been to make al-Qaeda look increasingly irrelevant. It played no part in the Arab spring - indeed, it was wrong-footed by it. Most of the grievances from which the network derived its support have since been addressed in the countries where regime change has taken place and Islamist parties have been elected freely without any threat of violence. It may turn out that the attacks of 11 September 2001, far from signalling the beginning of a new threat of global jihad, rather represented the high-water mark of al-Qaeda influence.
The Muslim Brotherhood in Tunisia, Egypt, Morocco and Syria is taking a moderate stand. Rather than look to Hamas as a role model, it is looking increasingly to Turkey's AKP (the centre-right Justice and Development Party), led by Prime Minister Recep Tayyip Erdogan. The Muslim Brothers see a religiously devout organisation that has been successful economically and politically, and many of them are keen to emulate that success. In fact, one of the aspects of the new world order to emerge most clearly out of the Arab spring is Turkey's leading role in the region, a phenomenon sometimes facetiously described as the "re-Ottomanisation" of the Arab world. In a recent opinion poll in five Arab countries, Turkey was seen as the country that has played the "most constructive role" in Arab events. In all five countries, the people said that after elections they wanted their president to look like Erdogan and their country to look like Turkey.
For the west, the Arab spring is a significant challenge and one to which we should rise - but we should not exaggerate our importance in events. The decisions will be taken by the peoples themselves, and regional powers such as Turkey and Saudi Arabia - and even minnows such as Qatar - will have more influence than we do.
We have, however, good reason to care about what happens. Prophylactically, we don't want a series of failed states close to Europe gestating terrorism or violence, as with the piracy in Somalia; we don't want extremism to spread as it has done from al-Qaeda in the Maghreb to Boko Haram in Nigeria; and we don't want unceasing waves of boat people arriving in Italy, Malta and Spain. More positively, we have an opportunity to contribute to the reformation of a part of the world that has a significant impact on our lives.
There is every reason to believe that the Middle East can make the journey from dictatorship to democracy and a market economy in the same way as Latin America has done from the 1970s to today. It won't be easy but we should not be neutral. We have to help. And there is a clear message to dictators and despots in the rest of the world, shaken by the Arab spring - they face the same choice as those in the Middle East did: evolution or revolution.
Jonathan Powell was chief of staff to Tony Blair from 1994-2007 and is the author most recently of "The New Machiavelli: How to Wield Power in the Modern World" (Vintage, £8.99)
This is an edited version of an essay that will appear in the forthcoming edition of PPR, the journal of the think tank IPPR