In 1843, shortly after his return from Afghanistan, an army chaplain, Reverend G R Gleig, wrote a memoir about the First Anglo-Afghan War, of which he was one of the very few survivors. It was, he wrote, "a war begun for no wise purpose, carried on with a strange mixture of rashness and timidity, brought to a close after suffering and disaster, without much glory attached either to the government which directed, or the great body of troops which waged it. Not one benefit, political or military, has Britain acquired with this war. Our eventual evacuation of the country resembled the retreat of an army defeated."

It is difficult to imagine the current military adventure in Afghanistan ending quite as badly as the First Afghan War, an abortive experiment in Great Game colonialism that slowly descended into what is arguably the greatest military humiliation ever suffered by the west in the Middle East: an entire army of what was then the most powerful military nation in the world utterly routed and destroyed by poorly equipped tribesmen, at the cost of £15m (well over £1bn in modern currency) and more than 40,000 lives. But nearly ten years on from Nato's invasion of Afghanistan, there are increasing signs that Britain's fourth war in the country could end with as few political gains as the first three and, like them, terminate in an embarrassing withdrawal after a humiliating defeat, with Afghanistan yet again left in tribal chaos and quite possibly ruled by the same government that the war was launched to overthrow.

Certainly it is becoming clearer than ever that the once-hated Taliban, far from being swept away by General Stanley McChrystal's surge, are instead regrouping, ready for the final act in the history of Hamid Karzai's western-installed puppet government. The Taliban have now advanced out of their borderland safe havens to the very gates of Kabul and are surrounding the capital, much as the US-backed mujahedin once did to the Soviet-installed regime in the late 1980s. Like a rerun of an old movie, all journeys by non-Afghans out of the capital are once again confined largely to tanks, military convoys and helicopters. The Taliban already control more than 70 per cent of the country, where they collect taxes, enforce the sharia and dispense their usual rough justice. Every month, their sphere of influence increases. According to a recent Pentagon report, Karzai's government has control of only 29 out of 121 key strategic districts.

Just recently, on 17 May, there was a suicide attack on a US convoy in the Dar-ul Aman quarter of Kabul, killing 12 civilians and six American soldiers; the following day, there was a daring five-hour-long grenade and machine-gun assault on the US military headquarters at Bagram Airbase, killing an American contractor and wounding nine soldiers, so bringing the death toll for US armed forces in the country to more than 1,000. Then, over the weekend of 22-23 May, there was a series of rocket, mortar and ground assaults on Kandahar Airbase just as the British ministerial delegation was about to visit it, forcing William Hague and Liam Fox to alter their schedule. Since then, a dozen top Afghan officials have been assassinated in Kandahar, including the city of Kandahar's deputy mayor. On 7 June, the deadliest day for Nato forces in months, ten soldiers were killed. Finally, it appears that the Taliban have regained control of the opium-growing centre of Marjah in Helmand Province, only three months after being driven out by McChrystal's forces amid much gung-ho cheerleading in the US media. Afghanistan is going down.

Already, despite the presence of huge numbers of foreign troops, it is now impossible – or at least extremely foolhardy – for any westerner to walk around the capital, Kabul, without armed guards; it is even more inadvisable to head out of town in any direction except north: the strongly anti-Taliban Panjshir Valley, along with the towns of Mazar-e-Sharif and Herat, are the only safe havens left for westerners in the entire country. In all other directions, travel is possible only in an armed convoy.

This is especially true of the Khord-Kabul and Tezeen passes, immediately to the south of Kabul, where as many as 18,000 British troops were lost in 1842, and which are today again a centre of resistance against perceived foreign occupiers. Aid workers familiar with Afghanistan over several decades say the security situation has never been worse. Ideas much touted only a few years ago that Afghanistan might become a popular tourist destination – a Switzerland of central Asia – now seem to be dreams from a distant age. Lonely Planet's guidebook to Afghanistan, optimistically published in 2005, has not been updated and is now once again out of print.

The present war is following a trajectory that is beginning to feel unsettlingly familiar to students of the Great Game. In 1839, the British invaded Afghanistan on the basis of sexed-up intelligence about a non-existent threat: information about a single Russian envoy to Kabul was manipulated by a group of ambitious and ideologically driven hawks to create a scare – in this case, about a phantom Russian invasion – thus bringing about an unnecessary, expensive and entirely avoidable war.

Initially, the hawks were triumphant – the British conquest proved remarkably easy and bloodless; Kabul was captured within a few weeks as the army of the previous regime melted into the hills, and a pliable monarch, Shah Shuja, was successfully placed on the throne. For a few months the British played cricket, went skating and put on amateur theatricals as if on summer leave in Simla; there were discussions about making Kabul the summer capital of the Raj. Then an insurgency began and that first heady success slowly unravelled, first among the Pashtuns of Kandahar and Helmand Provinces. It slowly gained momentum, moving northwards until it reached Kabul, so making the British occupation impossible to sustain.

What happened next is a warning of how bad things could yet become: a full-scale rebellion against the British broke out in Kabul, and the two most senior British envoys, Sir Alexander Burnes and Sir William Macnaghten, were assassinated, one hacked to death by a mob in the streets, the other stabbed and shot by the resistance leader Wazir Akbar Khan during negotiations. It was on the retreat that followed, on 6 January 1842, that the 18,000 East India Company troops, and maybe half that many again Indian camp followers, were slaughtered by Afghan marksmen waiting in ambush amid the high passes, shot down as they trudged through the icy depths of the Afghan winter. After eight days on the death march, the last 50 survivors made their final stand at the village of Gandamak. As late as the 1970s, fragments of Victorian weaponry and military equipment could be found lying in the screes above the village. Even today, the hill is said to be covered with the bleached bones of the British dead.

One Englishman lived to tell the tale of that last stand (if you discount the fictional survival of Flashman) – an ordinary foot soldier, Thomas Souter, wrapped his regimental colours around him to prevent them being captured, and was taken hostage by the Afghans who assumed that such a colourfully clothed individual must command a high ransom. It is a measure of the increasingly pertinent parallels between the 19th-century war and today's that one of the main Nato bases in Afghanistan was recently named Camp Souter after that survivor.

In the years that followed, the British defeat in Afghanistan became pregnant with symbolism. For the Victorian British, it was the country's greatest imperial disaster of the 19th century. It was exactly a century before another army would be lost, in Singapore in 1942. Yet the retreat from Kabul also became a symbol of gallantry against the odds: William Barnes Wollen's celebrated oil painting The Last Stand of the 44th Regiment at Gundamuck – showing a group of ragged but doggedly determined British soldiers standing encircled behind a porcupine of bayonets, as the Pashtun tribesmen close in – became one of the best-known images of the era, along with Remnants of an Army, Elizabeth Butler's image of the wounded and bleeding army surgeon William Brydon, who had made it through to the safety of Jalalabad, arriving before the city walls on his collapsing nag.

For the Afghans, the British defeat of 1842 became a symbol of freedom from foreign invasion. It is again no accident that the diplomatic quarter of Kabul is named after the general who oversaw the rout of the British in that year: Wazir Akbar Khan.

For south Asians, who provided most of the cannon fodder – the foot soldiers and followers killed on the retreat – the war ironically became a symbol of possibility: although thousands of Indians died on the march, it showed that the British army was not invincible and a well-planned insurgency could force them out. Thus, in 1857, the Indians launched their own anti-colonial uprising, the Great Mutiny (as it is known in Britain) or the first war of independence (as it is known in India), partly inspired by what the Afghans had achieved in 1842.

This destabilising effect on south Asia of the failed war in Afghanistan has a direct parallel in the blowback that is today destabilising Pakistan and the tribal territories of the Federally Administered Tribal Areas (Fata). Here the Pakistani Taliban are once more on the march, rebuilding their presence in Swat, and are now surrounding Peshawar, which is almost daily being rocked by bombs, while outlying groups of Taliban are again spreading their influence into the valleys leading towards Islamabad. Across much of the North-West Frontier Province – roughly a fifth of Pakistan's territory – women have now been forced into the burqa, music has been silenced, barbershops are forbidden to shave beards and more than 125 girls' schools have been blown up or burned down.

A significant proportion of the Peshawar elite, along with the city's musicians, have decamped to the relatively safe and tolerant confines of Lahore and Karachi, while tens of thousands of ordinary people from the surrounding hills of the semi-autonomous Fata tribal belt, and especially the Bajaur Agency (or tribal area), have fled from the conflict zones blasted by US Predator drones and strafed by Pakistani helicopter gunships to the tent camps ringing the provincial capital.

The Fata, it is true, have never been fully under the control of any Pakistani government, and have always been unruly, but the region has been radicalised as never before by the rain of shells and cluster bombs that have caused huge civilian casualties and daily add a stream of angry foot soldiers to the insurgency. Elsewhere in Pakistan, anti-western religious and political extremism continues to flourish, as ever larger numbers of ordinary Pakistanis are driven to fight by corruption, predatory politics and the abuse of power by Pakistan's feudal elite, as well as the military aggression of the drones. Indeed, the ripples of instability lapping out from Afghanistan and Pakistan have reached even New York. When CIA interrogators asked Faisal Shahzad why he tried to let off a car bomb last month in Times Square, he told them of his desire to avenge those "innocent people being hit by drones from above".

The route of the British retreat of 1842 backs on to the mountain range that leads to Tora Bora and the Pakistan border, an area that has always been a Taliban centre. I had been advised not to attempt to visit the area without local protection, and so last month I set off for the mountains in
the company of a regional tribal leader who was also a minister in Karzai's government. He is a mountain of a man named Anwar Khan Jegdalek, a former village wrestling champion who made his name as a Hezb-e-Islami mujahedin commander in the jihad against the Soviets in the 1980s.

It was Anwar Khan Jegdalek's ancestors who inflicted some of the worst casualties on the British army of 1842, something he proudly repeated several times as we drove through the same passes. "They forced us to pick up guns to defend our honour," he said. "So we killed every last one of those bastards." None of this, incidentally, has stopped Anwar Khan Jegdalek from sending his family away from Kabul to the greater safety of Northolt, Middlesex.

He drove himself in a huge 4x4, while a pick-up full of heavily armed Afghan bodyguards followed behind. We left Kabul – past the blast walls of the Nato barracks built on the very site of the British cantonment of 170 years ago – and headed down a corkscrewing road into the line of bleak mountain passes that links Kabul with the Khyber Pass.

It is a dramatic and violent landscape: fault lines of crushed and tortured strata groaned and twisted in the gunpowder-coloured rock walls rising on either side of us. Above, the jagged mountain tops were veiled in an ominous cloud of mist. As we drove, Anwar Khan Jegdalek complained bitterly of western treatment of his government. "In the 1980s when we were killing Russians for them, the Americans called us freedom fighters," he muttered, as we descended through the first pass. "Now they just dismiss us as warlords."

At Sorobi, where the mountains debouche into a high-altitude ochre desert dotted with encampments of nomads, we left the main road and headed into Taliban territory. A further five trucks full of Anwar Khan Jegdalek's old mujahedin fighters, all brandishing rocket-propelled gren­ades and with faces wrapped in keffiyehs, ­appeared from a side road to escort us.

At the crest of Jegdalek village, on 12 January 1842, 200 frostbitten British soldiers found themselves surrounded by several thousand Pashtun tribesmen. The two highest-ranking British soldiers, General Elphinstone and Brigadier Shelton, went off to negotiate but were taken hostage. Only 50 infantrymen managed to break out under cover of darkness. Our own welcome was, thankfully, somewhat warmer. It was my host's first visit to his home since he had become a minister, and the proud villagers took their old commander on a nostalgia trip through hills smelling of wild thyme and rosemary, and up on to mountainsides carpeted with hollyhocks, mulberries and white poplars. Here, at the top of the surrounding peaks, lay the remains of Anwar Khan Jegdalek's old mujahedin bunkers and entrenchments. Once the tour was completed, the villagers fed us, Mughal style, in an apricot orchard: we sat on carpets under a trellis of vine and pomegranate blossom as course after course of kebabs and mulberry pulao was laid in front of us.

During lunch, as my hosts casually pointed out the various places in the village where the British had been massacred in 1842, I asked them if they saw any parallels between that war and the present situation. "It is exactly the same," said Anwar Khan Jegdalek. "Both times the foreigners have come for their own interests, not for ours. They say, 'We are your friends, we want democracy, we want to help.' But they are lying."

“Whoever comes to Afghanistan, even now, they will face the fate of Burnes, Macnaghten and Dr Brydon," said Mohammad Khan, our host in the village and the owner of the orchard where we were sitting. The names of the fighters of 1842, long forgotten in their home country, were still known here.

“Since the British went, we've had the Russians," said an old man to my right. "We saw them off, too, but not before they bombed many of the houses in the village." He pointed at a ridge of ruined mud-brick houses.

“We are the roof of the world," said Mohammad Khan. "From here, you can control and watch everywhere."

“Afghanistan is like the crossroads for every nation that comes to power," agreed Anwar Khan Jegdalek. "But we do not have the strength to control our own destiny – our fate is always determined by our neighbours. Next, it will be China. This is the last days of the Americans."

I asked if they thought the Taliban would come back. "The Taliban?" said Mohammad Khan. "They are here already! At least after dark. Just over that pass." He pointed in the direction of Gandamak and Tora Bora. "That is where they are strongest."

It was nearly five in the afternoon before the final flaps of nan bread were cleared away, by which time it had become clear that it was too late to head on to the site of the British last stand at Gandamak. Instead, that evening we went to the relative safety of Jalalabad, where we discovered we'd had a narrow escape: it turned out there had been a huge battle at Gandamak that morning between government forces and a group of villagers supported by the Taliban. The sheer scale and length of the feast had saved us from walking straight into an ambush. The battle had taken place on exactly the site of the British last stand.

The following morning in Jalalabad, we went to a jirga, or assembly of tribal elders, to which the greybeards of Gandamak had come under a flag of truce to discuss what had happened the day before. The story was typical of many I heard about the current government, and revealed how a mixture of corruption, incompetence and insensitivity has helped give an opening for the return of the once-hated Taliban.

As Predator drones took off and landed incessantly at the nearby airfield, the elders related how the previous year government troops had turned up to destroy the opium harvest. The troops promised the villagers full compensation, and were allowed to burn the crops; but the money never turned up. Before the planting season, the villagers again went to Jalalabad and asked the government if they could be provided with assistance to grow other crops. Promises were made; again nothing was delivered. They planted poppy, informing the local authorities that if they again tried to burn the crop, the village would have no option but to resist. When the troops turned up, about the same time as we were arriving at nearby Jegdalek, the villagers were waiting for them, and had called in the local Taliban to assist. In the fighting that followed, nine policemen were killed, six vehicles destroyed and ten police hostages taken.

After the jirga was over, one of the tribal elders came over and we chatted for a while over a glass of green tea. "Last month," he said, "some American officers called us to a hotel in Jalalabad for a meeting. One of them asked me, 'Why do you hate us?' I replied, 'Because you blow down our doors, enter our houses, pull our women by the hair and kick our children. We cannot accept this. We will fight back, and we will break your teeth, and when your teeth are broken you will leave, just as the British left before you. It is just a matter of time.'"

What did he say to that? “He turned to his friend and said, 'If the old men are like this, what will the younger ones be like?' In truth, all the Americans here know that their game is over. It is just their politicians who deny this."

The defeat of the west's latest puppet government on the very same hill of Gandamak where the British came to grief in 1842 made me think, on the way back to Kabul, about the increasingly close parallels between the fix that Nato is in and the one faced by the British 170 years ago.

Now as then, the problem is not hatred of the west, so much as a dislike of foreign troops swaggering around and making themselves odious to the very people they are meant to be helping. On the return journey, as we crawled back up the passes towards Kabul, we got stuck behind a US military convoy of eight Humvees and two armoured personnel carriers in full camouflage, all travelling at less than 20 miles per hour. Despite the slow speed, the troops refused to let any Afghan drivers overtake them, for fear of suicide bombers, and they fired warning shots at any who attempted to do so. By the time we reached the top of the pass two hours later, there were 300 cars and trucks backed up behind the convoy, each one full of Afghans furious at being ordered around in their own country by a group of foreigners. Every day, small incidents of arrogance and insensitivity such as this make the anger grow.

There has always been an absolute refusal by the Afghans to be ruled by foreigners, or to accept any government perceived as being imposed on the country from abroad. Now as then, the puppet ruler installed by the west has proved inadequate to the job. Too weak, unpopular and corrupt to provide security or development, he has been forced to turn on his puppeteers in order to retain even a vestige of legitimacy in the eyes of his people. Recently, Karzai has accused the US, the UK and the UN of orchestrating a fraud in last year's elections, described Nato forces as "an army of occupation", and even threatened to join the Taliban if Washington kept putting pressure on him. Shah Shuja did much the same thing in 1842, towards the end of his rule, and was known to have offered his allegiance and assistance to the insurgents who eventually toppled and beheaded him.

Now as then, there have been few tangible signs of improvement under the western-backed regime. Despite the US pouring approximately $80bn into Afghanistan, the roads in Kabul are still more rutted than those in the smallest provincial towns of Pakistan. There is little health care; for any severe medical condition, patients still have to fly to India. A quarter of all teachers in Afghanistan are themselves illiterate. In many areas, district governance is almost non-existent: half the governors do not have an office, more than half have no electricity, and most receive only $6 a month in expenses. Civil servants lack the most basic education and skills.

This is largely because $76.5bn of the $80bn committed to the country has been spent on military and security, and most of the remaining $3.5bn on international consultants, some of whom are paid in excess of $1,000 a day, according to an Afghan government report. This, in turn, has had other negative effects. As in 1842, the presence of large numbers of well-paid foreign troops has caused the cost of food and provisions to rise, and living standards to fall. The Afghans feel they are getting poorer, not richer.

There are other similarities. Then as now, the war effort was partially privatised: it was not so much the British army as a corp­oration, the East India Company, that provided most of the troops who fought the war for Britain in 1842, just as today both the British and the Americans have subcontracted much of their security work to private companies. When I visited the British embassy, I found that many of the security guards at the gatehouse were not army or military police, but from Group 4 Security. The US security contracts offered to Blackwater/Xe and other private security forces under Dick Cheney's ideologically driven policy of privatising war are worth many millions of dollars.

Finally, now as then, there has been an attempt at a last show of force in order to save face before withdrawal. As happened in 1842, it has achieved little except civilian casualties and the further alienation of the Afghans. As one of the tribal elders from Jegdalek said to me: "How many times can they apologise for killing our innocent women and children and expect us to forgive them? They come, they bomb, they kill us and then they say, 'Oh, sorry, we got the wrong people.' And they keep doing that."

The British soldiers of 1842 found the same reaction in their day. In his diary of his time with the British army of retribution, which laid waste to great areas of southern Afghanistan as punishment for the massacres on the retreat from Kabul earlier in the year, the young Captain N Chamberlain reported how his troops inflicted horrible atrocities on any Afghan civilians they could find. One morning he met a wounded Afghan woman dragging herself towards a stream with a water pot. "I filled the vessel for her," he wrote, "but all she said was, 'Curses on the feringhees [foreigners]!' I continued on my way disgusted with myself, the world, above all with my cruel profession. In fact, we are nothing but licensed assassins."

However, there are some important differences between Britain's first defeat in Afghanistan and the current mess. In 1842, we were at least reinstalling a legitimate Afghan ruler and removing one who could genuinely be cast as an illegitimate usurper. Shah Shuja, the British puppet, was a former ruler of the Sadozai dynasty, from the leading Pashtun clan, and a grandson of the great Ahmed Shah Durrani, the first king of a united Afghanistan. As the traveller and pioneering archaeologist Charles Masson observed: "The Afghans had no objection to the match; they merely disliked the manner of the wooing."

This time, we have been clumsier, and Nato has helped instal a former CIA asset accused by a high-ranking UN diplomat of drug abuse and of having a history of mental instability, with little to recommend him other than that he was once run out of Langley. Although Karzai is a Pashtun of the Popalzai tribe, under his watch Nato has in effect installed the Northern Alliance in Kabul and driven the country's Pashtun majority out of power.

The reality of our present Afghan entanglement is that we took sides in a complex civil war, which has been running since the 1970s, siding with the north against the south, town against country, secularism against Islam, and the Tajiks against the Pashtuns. We have installed a government, and trained up an army, both of which in many ways have discriminated against the Pashtun majority, and whose top-down, highly centralised constitution allows for remarkably little federalism or regional representation. However much western liberals may dislike the Taliban – and they have very good reason for doing so – the truth remains that they are in many ways the authentic voice of rural Pashtun conservatism, whose views and wishes are ignored by the government in Kabul and who are still largely excluded from power. It is hardly surprising that the Pashtuns are determined to resist the regime and that the insurgency is widely supported, especially in the Pashtun heartlands of the south and east.

Yet it is not too late to learn some lessons from the mistakes of the British in 1842. Then, British officials in Kabul continued to send out despatches of delusional optimism as the insurgents moved ever closer to Kabul, believing that there was a straightforward military solution to the problem and that if only they could recruit enough Afghans to their army, they could eventually march out, leaving that regime in place – exactly the sentiments expressed by the Defence Secretary, Liam Fox, on his visit to Afghanistan last month.

In 1842, by the time they realised they had to negotiate a political solution, their power had ebbed too far, and the only thing the insurgents were willing to negotiate was an unconditional surrender. Today, too, there is no easy military solution to Afghanistan: even if we proceed with the plan to equip an army of half a million troops (at the cost of roughly $2bn a year, when the entire revenue of the Afghan government is $1.1bn – in other words, 180 per cent of revenue), that army will never be able to guarantee security or shore up such a discredited regime. Every day, despite the military power of the US and Nato and the $25bn so far ploughed into rebuilding the Afghan army, security gets worse, and the area under government control contracts week by week.

The only answer is to negotiate a political solution while we still have enough power to do so, which in some form or other involves talking to the Taliban. This is a course that Karzai, to his credit, is keen to pursue; he made it clear that his peace jirga at the start of this month was open to any Taliban leader willing to lay down arms, and that jobs and monetary incentives would be available to former Taliban who changed their allegiance and joined the government. It is still unclear whether the new Tory government supports this course; Barack Oba­ma certainly opposes it. In this, he is supported by the notably undiplomatic US envoy to the region, Richard Holbrooke, described by one senior British diplomat as "a bull who brings his own china shop wherever he goes".

There is something else we can still do before we pull out: leave some basic infrastructure behind, a goal we notably failed to achieve in the past nine years. Yet William Hague and Liam Fox oppose this policy – as Fox notoriously said in his 21 May interview with the Times, which infuriated his Afghan hosts: "We are not in Afghanistan for the sake of the education policy in a broken 13th-century country." The Tories could do much worse than consult their own newly elected backbencher Rory Stewart. He knows much more about Afghanistan than either Fox or Hague. As Stewart wrote shortly before he entered politics, targeted aid projects that employ Afghans can do a great deal of good, "and we should focus on meeting the Afghan government's request for more investment in agriculture, irrigation, energy and roads".

In the meantime, Obama has announced that he will begin withdrawing troops in July 2011. The start of the US withdrawal is likely to begin a rush to evacuate the other Nato forces located in pockets around the country: the Dutch have announced that they will be pulling out of Uruzgan this summer, and the Canadian and Danes won't be far behind them. Nor will the Brits, despite assurances from Hague and Fox. A recent poll showed that 72 per cent of Britons want their troops out of Afghanistan immediately, and there is only so long any government can hold out against such strong public opinion. Certainly, it is time to shed the idea that a pro-western puppet regime that excludes the Pashtuns can remain in place indefinitely. The Karzai government is crumbling before our eyes, and if we delude ourselves that this is not the case, we could yet face a replay of 1842.

George Lawrence, a veteran of that war, issued a prescient warning in the Times just before Britain blundered into the Second Anglo-Afghan War in the 1870s. "A new generation has arisen which, instead of profiting from the solemn lessons of the past, is willing and eager to embroil us in the affairs of that turbulent and unhappy country," he wrote. "Although military disasters may be avoided, an advance now, however successful in a military point of view, would not fail to turn out to be as politically useless."

William Dalrymple's latest book, "Nine Lives: in Search of the Sacred in Modern India", won the first Asia House Literary Award in May, and is newly published in paperback (Bloomsbury, £8.99). His book on the First Anglo-Afghan War is planned for release in autumn 2012