It may have been a nightmarish year for Pakistan but it has been a pretty good one for the country's inscrutable chief of army staff, the most powerful man in the Land of the Pure, General Ashfaq Parvez Kayani.

For a start, the army's response to the floods has compared well to the usual corrupt incompetence of Pakistan's civilian politicians, guided by their chateau-hopping president, Asif Ali Zardari (while minister for investment, he was nicknamed "Mr 10 Per Cent"; he has now been upgraded to "Mr 110 Per Cent"). This has led to discussion in army circles about whether it is time to drop the civilian fig leaf and return the country to the loving embrace of its military. So serious is this threat, that one of the country's most senior and well-connected journalists, Najam Sethi, editor-in-chief of the Friday Times, went on the record this month to warn that elements in the army were plotting yet another coup. "I know this is definitely being discussed," he said.

Then there was the news that Kayani was going to be allowed to keep his job for a second term: "an extraordinary situation requires an extraordinary decision to overcome it", explained a brigadier, writing in the Nation newspaper. Kayani, a former head of Pakistan's notorious intelligence agency, the Inter-Services Intelligence (ISI), can now continue to run the army, and by default Pakistan's foreign policy, until November 2013.

But Kayani's biggest triumph this year, arguably the greatest of his career, was his visit to Kabul in July as the honoured guest of the Afghan president, Hamid Karzai. The visit marked an important thawing in Pak-Afghan relations, which have been glacial ever since Karzai came to power in 2001. It also coincided with the sacking of Amrullah Saleh, Karzai's pro-Indian and rabidly anti-ISI former security chief. Saleh is a tough Tajik who rose to prominence as a mujahedin protégé of Ahmed Shah Massoud, the Indian-backed "Lion of Panjshir". The Taliban, and their sponsors in the ISI, had regarded Saleh as their fiercest opponent, something Saleh was enormously proud of.

When I had dinner with him in Kabul in May, he spoke at length of his frustration with the ineffectiveness of Karzai's government in taking the fight to the Taliban, and the extent to which the ISI was managing to aid, arm and train its puppet insurgents in North Waziristan and Balochistan. Saleh's sacking gave notice of an important change of direction by Karzai. As Bruce Riedel, Barack Obama's Af-Pak adviser, said when the news broke, "it means that Karzai is already planning for a post-American Afghanistan".

It seems that Kayani and Karzai are discussing some sort of accommodation between the Afghan government and ISI-sponsored elements in the Taliban, maybe those of Sirajuddin Haqqani, which could give over much of the Pashtun south to pro-Pakistan Taliban, but preserve Karzai in power in Kabul after the US withdrawal. The expulsion of India, Pakistan's great regional rival, from Afghanistan, or at least the closing of its four regional consulates, would be a top priority for the ISI in return for any deal that kept Karzai in power.

With the US toppling of the Taliban after the 11 September 2001 attacks, Pakistan's influence disappeared abruptly from Afghanistan and
India quickly filled the vacuum. To the ISI's horror, in the early years of this decade, India provided reconstruction assistance and training worth roughly £835m in total. It also built roads, sanitation projects, the new Afghan parliament and free medical facilities across the country. It even offered to help train the Afghan army. Nato refused. As General Stanley McChrystal put it in a report last year, "while Indian activities largely benefit the Afghan people, increasing Indian influence in Afghanistan is likely to exacerbate regional tensions and encourage Pakistani countermeasures".

McChrystal was right. The Pakistanis have always been paranoid about the small Indian presence in Afghanistan. "We have strong evidence [that India is] using Afghanistan against Pakistan's interests to destabilise Pakistan," a foreign ministry spokesman claimed in March. Pakistan's military establishment, terrified of the economic superpower emerging to the south, believes it would be suicide to accept an Indian presence in what it regards as its Afghan backyard - a potential point of retreat for the army in the event of an Indian invasion, something Pakistani analysts refer to as vital "strategic depth".

According to Indian diplomatic sources, there are still fewer than 3,600 Indians in Afghan­istan; there are only ten Indian diplomatic officers, as opposed to nearly 150 in the UK embassy. Yet the horror of being encircled has led the ISI to risk Pakistan's relationship with its main strategic ally, the US, in order to keep the Taliban in play and its leadership under ISI patronage in Quetta - a policy Kayani developed while head of the ISI. Karzai's new deal with the Pakistanis, and his clear intention to try to reach some accommodation with their proxies among the Taliban, therefore represents a major strategic victory for Kayani and Pakistan's military, as well as a grave diplomatic defeat for India.

Pakistan's support for the Taliban today is only the most recent chapter of an old story of complicity between jihadi movements and the Pakistani state. Since the days of the anti-Soviet mujahedin, Pakistan's army saw violent Islamic groups as an ingenious and cost-effective means of both dominating Afghanistan (which they finally achieved with the retreat of the Soviets in 1987) and bogging down the Hindu-dominated Indian army in Kashmir (which they managed with great effect from 1990 onwards).

The former ISI director and Dick Dastardly lookalike Hamid Gul, who was largely responsible for developing the strategy, once said to me: "If the jihadis go out and contain India, tying down one million men of their army on their own soil, for a legitimate cause, why should we not support them?" Next to Gul in his Islamabad living room lay a piece of the Berlin Wall presented to him by the city's people for "delivering the first blow" to the Soviet empire through his use of jihadis in the 1980s. The WikiLeaks documents suggest he is still busy liaising with jihadis in his "retirement".

The Pakistani military top brass were long convinced that they could control the militants they have nurtured. In a taped conversation between President Pervez Musharraf and Muhammad Aziz Khan, his chief of general staff, that India released in 1999, Aziz said the army had the jihadis by their "tooti" (cock or neck). Yet the Islamists have increasingly followed their own agendas, sending suicide bombers out against not just Pakistan's religious minorities and political leaders, but even the ISI headquarters. Nonetheless, many in the army still believe the jihadis are a more practical defence against Indian hegemony than nuclear weapons. For them, supporting Islamist groups is not an ideological or religious whim, so much as a practical and patriotic imperative - a vital survival strategy for a Pakistani state.

The army and ISI continued this duplicitous and risky policy after 11 September 2001 despite Musharraf's public promises to the contrary. The speed with which the US lost interest in Afghanistan after its invasion and embarked on plans to invade Iraq convinced the Pakistani army that the Washington had no long-term commitment to Karzai's regime. This led to the generals keeping the Taliban in reserve, to be used to reinstal a pro-Pakistani regime in Kabul once the American gaze had turned elsewhere.

So it was that the ISI gave refuge to the leadership of the Taliban after it fled from Afghan­istan in 2001. Mullah Mohammed Omar was kept in an ISI safehouse in Quetta; his militia was lodged in the sprawling suburb of Pashtunabad. There, Mullah Abdul Ghani Baradar presided over the Taliban military committee and war chest. Gulbuddin Hekmatyar, the leader of Hizb-e-Islami, was lured back from exile in Iran and allowed to operate freely outside Peshawar, while Jalaluddin Haqqani, one of the most violent Taliban commanders, was given sanctuary in North Waziri­stan. Other groups were despatched to safehouses in Balochistan.

By 2004, the US had filmed Pakistani army trucks delivering Taliban fighters at the Afghan border and recovering them a few days later; wireless monitoring at the US base at Bagram picked up Taliban commanders arranging with Pakistani army officers at the border for safe passage as they came in and out of Afghanistan. Western intelligence agencies concluded that the ISI was running a full training programme for the Afghan Taliban, turning a blind eye as they raised funds in the Gulf and allowing them to import materiel, mainly via Dubai. By 2005 the Taliban, with covert Pakistani support, were launching a full-scale assault on Nato troops in Afghanistan and being given covering fire as they returned to their bases in Pakistan.

At the same time, Taliban attacks on Indian interests in Afghanistan intensified, beginning the process of turning the Afghan conflict, like that in Kashmir, into what it is today: an Indo-Pak proxy war. The Indian embassy in Kabul was twice bombed - in July 2008 and October 2009 - as were two city-centre hotels thought to have been used by the Research and Analysis Wing (Raw), the Indian intelligence agency. Seven Indian civilians and two Indian military officers died in the blasts.

The degree to which the ISI has been controlling the Afghan Taliban has only just become clear, and not just in the documents published by WikiLeaks. A report by Matt Wald­man of the Carr Centre for Human Rights at Harvard, based on interviews with ten former senior Taliban commanders, closely documents how the ISI "orchestrates, sustains and strongly influences" the Taliban and shows how the ISI is even represented on the Taliban's supreme leadership council, the Quetta Shura.

Meanwhile, in the Punjab, Hafiz Mohammad Saeed, founder of Lashkar-e-Toiba, and the man believed to have been behind the 2008 Bombay attacks, has been allowed to continue operating from Muridke, near Lahore. "The powerful western world is terrorising Muslims," he told a conference in Islamabad this year. "We are being invaded, manipulated and looted. We must fight the evil trio of America, Israel and India. Suicide missions are in accordance with Islam. In fact, a suicide attack is the best form of jihad."

At the same time as pursuing its policy of selectively using jihadis, Pakistan has appeased the US by giving generous assistance to the CIA in arresting foreign Arab al-Qaeda personnel. A major assault was also launched against both the militants who took over the Lal Masjid and the ultra-radical Pakistan Taliban who took over the Swat Valley and announced their intention of turning the country into an Islamic Emirate. In the course of these operations and others in the Federally Administered Tribal Areas, more than 1,500 Pakistani soldiers and policemen were killed; another 250,000 people were made homeless in the Pak army assault on Bajaur. The ISI has even been prepared to arrest any members of the Afghan Taliban who didn't follow orders. Hence the seizure in Karachi, in February, of the Taliban second-in-command, Mullah Abdul Ghani Baradar, along with about a dozen other senior Taliban whom the ISI regarded as infringing on their hospitality by opening talks with the Karzai administration via the Saudis, without ISI clearance.

Yet, even though the Pakistani army has conducted major offensives in six of the tribal areas, the seventh, North Waziristan, has been left alone, as it is home to the ISI's favourite proxies: Haqqani and Hekmatyar. Similarly, Pakistan's foot-dragging response to the 2008 attacks on Bombay, and the lack of response to the attacks on minority faith groups in Pakistan over the past few months, show that the Janus-faced policy remains in place. This summer, the chief minister of the Punjab, Shahbaz Sharif, was quoted heatedly denying that there were any militant groups working in the Pakistani Punjab, or that the Punjabi Taliban even existed. There are still, in the eyes of many in the Pakistani establishment, good Taliban and bad Taliban, useful militants and expendable ones.

In their eyes, the ongoing defeat of Nato in Afghanistan, with US and British troops suffering record casualties last month, is a vindication of its long-term strategy. Islamabad has succeeded in regaining influence in Afghanistan and Delhi has been checked. But India will not take this lying down. Already the Indian press has reported attempts to resuscitate the Northern Alliance as a contingency against the Taliban's takeover of the south, and here India is working in conjuction with Russia, Iran and the central Asian "stans". The Indian national security adviser, Shivshankar Menon, was despatched to Afghanistan in March, and the foreign minister, S M Krishna, has visited Tehran. Post-American Afghanistan is looking increasingly likely to be divided between the Tajik, Uzbek and Hazara north and the Pashtun south, either formally, with a partition, or more likely, to slip into inter-ethnic civil war, with India supporting and arming the north and Pakistan the south.

As diplomacy gathers pace, the Afghan Tali­ban, who already control over 70 per cent of the country, continue to increase their power. The most worrying development has been the spread of Taliban units to the previously peaceful north, where they have taken over pockets of Pathan settlement around Kunduz and Badakhshan. The death of the British aid worker Karen Woo on 5 August was a direct result.

In Pakistan, too, jihadi activity is growing. Last year there were 87 suicide attacks across the country, killing roughly 3,000 people and the ISI this week stated that, for the first time in the nation's history, it regarded home-grown Islamic militants to be a bigger threat to the integrity of the nation than India. Yet the army continues to obsess about India. In a recent speech, Kayani emphasised that although the army knows the dangers of militancy, it was against Indian attacks that he was principally focused. At a time when Pakistan's economy is in crisis, electricity supply increasingly erratic and the educational system in complete breakdown, Kayani has secured a huge increase in the country's defence budget.

It is not a pretty picture: growing violence everywhere, increasing Indo-Pak tensions and a defeat for western interests in the region. Worst of all, because the Pakistani army regards this as a major triumph, it is unlikely to change its policy any time soon.

William Dalrymple's "Nine Lives: in Search of the Sacred in Modern India" won the first Asia House Literary Award in May and is newly published in paperback (Bloomsbury, £8.99)