Had Osama Bin Laden lived to see the present state of the Middle East he would have been rather pleased. The realisation of his ultimate ambition is gripping the Levant with the announcement of a caliphate straddling parts of Syria and Iraq. Controlling a piece of land roughly the size of Jordan and bigger than either Israel or Lebanon, Islamic State’s leader, Abu Bakr al-Baghdadi is demanding international attention unlike any of his predecessors.

Islamic State is perhaps the most aggressive invading force in the Levant since the Mongols. Moreover, it is being given a free hand to recast the contours of power in what remains one of the world’s most sensitive (and volatile) geostrategic locations. This is no accident. The implosion of both Syria and Iraq, coupled with western reluctance to intervene in what is seen as yet another Arabian calamity, has fuelled the sudden rise of Baghdadi’s millenarian militia.

This is precisely what Bin Laden always envisioned. His main thesis on the failure of the Islamist project was that western interference in the Middle East prevented the rise of Islamic governments. Weaken the west’s sphere of influence, he argued, and a caliphate would emerge.

Events helped crystallise this view. Shortly after the Afghan mujahedin’s unlikely victory over the Soviet Union in the late 1980s, Saddam Hussein invaded Kuwait and King Fahd turned to the United States to defend Saudi Arabia against his Ba’athist neighbour. Bin Laden was left embittered by the experience after the House of Saud scuppered his hopes of using the mujahedin to repel Saddam from Kuwait.

The humiliation for returning jihadists did not end there. Many from North Africa and the Gulf found themselves imprisoned and persecuted on their return. It was soon clear that going home was not an option and many of the Afghan alumni subsequently began to congregate in Sudan under the patronage of the chairman of the ruling party, Hassan al-Turabi, who had formed a Sunni Islamist movement at the time.

For the Arab fighters it was a comedown from their intoxicating victories in the mountains of the Hindu Kush against one of the great superpowers.

In Sudan, these fighters largely continued pursuing Islamo-nationalist aims. The Egyptians focused on Egypt, the Algerians on Algeria, and the Libyans on Libya. However, Saudi Arabia captured everyone’s attention. The arrival of US troops in the Arabian Peninsula – home to Islam’s most holy sites and regarded as sacred soil by Islamists – assaulted the imagination. This is when the gear shift occurred, redirecting the focus of jihadist anger from the metropolis to the periphery.

In an interview with the London-based Arabic-language newspaper al-Quds al-Arabi in 1996, Bin Laden explained: “. . . we believe that the US government committed the biggest mistake when it entered a peninsula which no non-Muslim nation has ever entered for 14 centuries . . . [America’s] entry was arbitrary and a reckless action. They have entered into a confrontation with a nation whose population is one billion Muslims.”

Having settled in Sudan, Bin Laden campaigned for Islamic revival in Saudi Arabia by establishing the Committee for Advice and Reform. This organisation had registered offices in Holborn, London, and was led by another veteran of the Afghan campaign, Khaled al-Fawwaz, who acted as Bin Laden’s representative in London.

Between 1994 and 1995 Bin Laden used his London address to send a total of 14 letters to the Saudi government. All of these urged the Saudi state to end co-operation with the United States. What he wanted instead was a more isolationist and self-assured form of Islam – a purer interpretation of sharia law, an end to western economic influence and a more Muslim-centred foreign policy.

Another letter by Bin Laden to King Fahd explained: “It is not reasonable to keep silent about the transformation of our nation into an American protectorate which is defiled by the soldiers of the Cross with their impure feet in order to protect your crumbling throne and preserve oilfields in the kingdom.” He continued:

Is it not right for the [Islamic] nation to wonder about who is behind instability and turbulence in the country? Is it the system that delivered the country into a state of chronic military debilitation in order to justify bringing the Jewish and Christian forces to defile the holy lands? Or is it the people who call for the preparation of the nation, arming it to be strong enough to take matters into its hands, protecting its honour and religion, defending its holy sites, its land and dignity?

Fawwaz sent all of these letters on Bin Laden’s behalf until he was indicted by the US for his alleged involvement in the 1998 bombings of the US embassies in Kenya and Tanzania. Bin Laden himself was implicated in the attacks.

Although a rationale of revenge was the primary argument Bin Laden put forward to justify the 11 September 2001 attacks, he also argued that confronting the US directly would undermine and weaken Arab regimes back home. Indeed, this is how al-Qaeda has sought to credit itself with the Arab-world uprisings of 2011, otherwise referred to as the “Arab spring”.

“The abandonment of America’s allies one by one is the fallout of its diminishing pride and arrogance after receiving the blows in New York, Washington and Pennsylvania,” argued Ayman al-Zawahiri, Bin Laden’s second-in-command, shortly after the fall of Hosni Mubarak in Egypt in February 2011. The 9/11 attacks had “directly caused America to lose influence over the [Arab] people because its grasp over the [Arab] regimes was weakened”.

Fantastical though such a view may be, it explains al-Qaeda’s grand strategy for effecting change.

****

Nowhere was the policy of direct confrontation with the US more apparent than in Iraq. Led by Abu Musab al-Zarqawi, al-Qaeda in Iraq launched a deliberately brutalising campaign aimed at shocking the west. From Iraq, Zarqawi sought to traumatise western societies into ever more reticence about intervention. His campaign struck directly at those who had supported “Operation Iraqi Freedom”, claiming 4,486 American lives in the process and a further 318 from allied forces. The civilian death toll was immeasurably higher.

Traumatising as these casualties were, it is the broader cultural ramifications of the conflict that have left an indelible scar on both our society and politicians. Large sections of the Arab world – not just those already consumed with a deep suspicion of the west – erupted in a fit of anti-Americanism after the Iraq war. Every death of a western solider was cheered, every suicide bombing applauded; a Nelsonian eye was turned to the excesses of al-Qaeda in Iraq.

This perversion enveloped the entire region, from the trendy guests at Lebanese beach parties to the chattering classes of Dubai’s bevelled hotel lobbies. It is this cultural disengagement by ordinary Arabs, otherwise wholly unaligned to jihadist groups, that has proved so shocking.

While the west recorded uneven results in Iraq, the campaign was broadly a success for the global jihad movement. Zarqawi not only achieved a small foothold for his fighters in Iraq but also successfully redefined the balance of power within al-Qaeda. By 2005, his brutal campaign across Iraq had begun to alienate much of the regional support al-Qaeda previously enjoyed. This worried the central leadership. Ayman al-Zawahiri wrote to Zarqawi, chastising him for two things in particular: executing hostages and pursuing a bloody, sectarian conflict with the Shias. “Many of your Muslim admirers among the common folk are wondering about your attacks,” he wrote. “Don’t lose sight of the target.”

The overtures had no impact. Zarqawi rebuked Zawahiri by insisting that he was on the ground and therefore best placed to decide what strategy the group should pursue. This prompted a lasting shift in the internal dynamics of the jihad movement – proximity now confers legitimacy. Those on the periphery could never be better placed than Zarqawi to dictate the prevailing strategy.

That precedent directly fuelled the rise of Islamic State today. Since Zarqawi’s death in 2006, al-Qaeda in Iraq has drifted into greater autonomy, renaming itself as the Islamic State of Iraq (ISI) that year. Although still nominally tied to al-Qaeda, the ISI was a largely independent group.

Relations finally unravelled with the onset of the Syrian civil war. Syrian fighters from ISI led by Abu Muhammad al-Jawlani moved back into the country and established Jabhat al-Nusrah. They were supposed to serve as al-Qaeda’s official representatives on the ground, though ISI could not resist direct involvement. Abu Bakr al-Baghdadi eventually ordered his own men into Syria, rebranding his group the Islamic State of Iraq and al-Sham (Isis) and ordering Jabhat al-Nusrah to disband.

Zawahiri was furious. He insisted that Baghdadi limit his ambitions to Iraq and leave the Syrian campaign to Jawlani. It was not only the Qaeda leader who suggested this. Notable jihadist ideologues from around the world echoed these sentiments, including Abu Qatada, the radical Muslim preacher who was deported from London back to Jordan last year.

The disagreement opened up a chasm in the global jihad movement. Both al-Qaeda and theoreticians associated with the group had urged Baghdadi to fall into line, only to be rebuffed. Invoking the primacy of proximity, as Zarqawi had done, spokesmen for Isis strongly rejected suggestions that the group was acting ultra vires. “The wars in Syria and Iraq are the same,” explained Abu Muhammed al-Adnani, a leading spokesman for Isis. In both cases, the group insists, it is protecting Sunni Islam against Shia forces.

What is significant is how Isis has sought to justify itself to the broader community of jihadi supporters. It is al-Qaeda and its ideologues – not Isis – that has betrayed the true spirit of what Osama Bin Laden always envisioned. And Isis is the rightful heir to his legacy, exploiting the power vacuum in the Levant to create an Islamic state.

The Isis leaders’ frustration is understandable. They regard the current US inaction in the region as stemming directly from the Americans’ confrontation with them during the Iraq war from 2003. The spectre of that engagement continues to cast a long and enveloping shadow over western societies. It is precisely what Bin Laden had predicted would happen, which makes Zawahiri’s reluctance to capitalise on it all the more inexplicable.

Withdrawing to Iraq would signal an acknowledgement of the boundaries set in the 1916 Sykes-Picot Agreement, a false aberration imposed on Muslims by “crusaders”. Moreover, Adnani accuses Zawahiri of prioritising politics over jihad. Only this could explain why al-Qaeda did not exploit the uprisings in Tunisia, Egypt and Libya. The corollary is clear: al-Qaeda has lost its way under Zawahiri.

In many senses, Islamic State has now surpassed al-Qaeda altogether. Whereas al-Qaeda is a terrorist organisation committed to confronting the west violently, Islamic State has grander ambitions. Once a terrorist group, it morphed into a sophisticated insurgency, and now operates its own state.

The organisation is also investing heavily in winning public support. It operates a broad range of social services, ensuring that people under its authority have access to basic necessities such as health care, education and fuel, as well as other public services. In July, during the Muslim festival of Eid, it hosted recreational events, including pie-eating contests for children and a tug of war for adults.

****

When Khaled al-Fawwaz came to London as Osama Bin Laden’s representative in the late 1980s he was just one of many Islamists pouring into the country. Others such as Abu Qatada, Abu Hamza and Omar Bakri Mohammed followed and, in the process, they established a sophisticated Islamist network across the UK.

In 1994 a major international conference promoting the caliphate was held in London, gathering radical clerics from around the world. Some early adherents of Islamism even went on to fight in Bosnia and Chechnya. Others pursued more esoteric aims in states such as Yemen.

It is telling to chart the evolution of British Islamist discourse through the 1990s. When the 1994 caliphate conference was convened, a large part of proceedings was dedicated to discussing what the caliphate is and whether it is obligated in Islam. By the end of the decade, the idea of the caliphate was entrenched and the debate moved on. What Muslims discussed then was precisely how – not whether – the caliphate should be revived. Seen in this way, it is clear that the roots of Islamist ideology run deep in some parts of British Muslim life.

The caliphate is a broad concept bound up with another set of ideas, too. At its core lies an alternative identity, the umma – a fraternity of the faithful, in which loyalty and allegiance are defined through confessional affiliation over civic ideals. It is the belief in the umma that has inspired as many as 500 British men (and a handful of women) to pack their bags and migrate to Syria.

British jihadists are not in Syria to melt into the background. They are full and fervent participants in the conflict. In the past 12 months British fighters have volunteered to be suicide bombers, executed prisoners of war, and tortured detainees in their care.

Fighters from groups as diverse as Jabhat al-Nusrah, Ahrar al-Sham and the Free Syrian Army have all told me of their concerns over the extremism of British jihadists. They are regarded as some of the most vicious and vociferous. The issue emerged in sharp relief this past week with the murder of the American journalist James Foley, seemingly by a British executioner with a London accent.

Pressure is mounting on the Prime Minister to address the flow of British fighters into groups such as Islamic State. The challenges are many, not least because there is a perception in some parts that the battle against Islamism has been won.

The main institutions promoting an Islamist agenda in this country – the Muslim Council of Britain, the Muslim Association of Britain and the Federation of Islamic Organisations in Europe, among others – have been beaten back. So, too, have the most prominent preachers of radical ideology. At the same time, hardline organisations such as al-Muhajiroun and Hizb ut-Tahrir have fallen into obscurity.

To an extent, it can be said that there has been a decline in Islamist agitation in the public sphere. Yet this success does not represent the whole picture. Because Islamist ideas have flourished in parts of the UK for more than two decades they remain a potent and pervasive force. That explains how a generation of men not even born during the 1994 caliphate conference has come to embrace the Syrian jihad so eagerly.

In 2011 David Cameron issued one of the clearest statements by any politician of the need to inflect British values in the public sphere. In what popularly became known as the “Munich speech”, the Prime Minister spoke of the need to build a strong civic identity to which all members of our society could subscribe. A few months later, Lord Carlile published his review into the Prevent counterterrorism strategy, adumbrating a new vision for the initiative.

Both the Prime Minister and Lord Carlile identified the role of Islamist ideology as a primary driver of radicalisation. It was a marked departure from the cosseted approach adopted by their Labour predecessors in government, though much work remains to be done in this regard.

Dissuading young men not to join jihadist organisations in Syria and Iraq is proving to be an arduous task. Where it was once thought that the domestic terrorism threat was being managed down, the revival of jihadist fortunes in Syria has extended its lifespan by another generation or two.

****

The Islamic State surge is not the first time a jihadist organisation has succeeded in taking swaths of land. It has happened before. Al-Qaeda and Taliban forces held significant parts of Pakistan’s North-West Frontier Province. In Somalia, al-Shabab has established itself in certain parts; Ansar Dine asserted itself over a significant area of northern Mali most recently.

What distinguishes Islamic State from its predecessors is that in every one of those cases there was international momentum to unseat the jihadists. Western coalition forces worked with Pakistan to uproot militants from the tribal areas, while the African Union Mission in Somalia (AMISOM), backed by the UN, pushed back al-Shabab. In Mali, the French committed ground troops to overcome Ansar Dine and its affiliates.

There is no comparable momentum arrayed against Islamic State. Neither the Iraqi nor the Syrian army is capable of overcoming it. Regional actors led by Saudi Arabia, Qatar and Turkey are unwilling to act and favour arming other rebel groups instead, a policy that has failed to deliver any meaningful results so far.

The western world looks on and sees only a conflict within Islam – Sunni pitted against Shia – and asks why we should intervene. The post-9/11 campaigns in Afghanistan and Iraq appear not to have been worthwhile. This cognitive dissonance has allowed Abu Bakr al-Baghdadi to revive a caliphate in the heart of the Arab and Muslim world.

But public opinion is beginning to take notice of Islamic State. With the execution of James Foley and the prominence of European (especially British) fighters in the conflict, it cannot be ignored. And yet, the belated approach of western policymakers has made Islamic State an entrenched entity. It is a state in every sense of the word. It maintains a treasury of billions of dollars, provides social services and has an army of skilled fighters with combat experience.

All this points to one conclusion, however depressing: Islamic State will not be overcome without some form of western military intervention. 

Shiraz Maher is a senior research fellow at the International Centre for the Study of Radicalisation at King’s College London and at Johns Hopkins University