Six years ago, in the autumn of 1998, I was sitting in a roadside tea shop surrounded by the desert landscape of Rajasthan, when I saw a succession of bicycle rickshaws appear over the horizon. There were five in all, and they were winding their way through the dusty scrub of the Jaipur highway. From the canopy of each rickshaw flew a small green flag on which was embroidered a silver crescent. As they came nearer, I saw that inside the rickshaws were 12 Sufi dervishes, with long unkempt beards. The drivers and dervishes were all hot and thirsty, and pulled into the dhaba calling for water and tea.
"Where are you going?" I asked, as the men began to step down and shake the desert from their clothes.
"To the Urs [festival] of Khwaja Garib Nawaz at Ajmer," said a driver. "We have bicycled these men all the way from Delhi."
"Delhi? But that is - what? - 400 kilometres?"
"Garib Nawaz [literally, 'The Friend of the Poor'] will reward us for our pains," said the second rickshaw driver. "It is he who gives us strength."
"Anyone who steps through the door of his shrine," said one of the dervishes, "will get paradise as his everlasting home."
I was heading in the same direction, so the following day I went along to the Sufi shrine in Ajmer. It was hosting one of the largest Muslim mystical festivals in the entire Islamic world. Virtually overnight, a small provincial town had been transformed into a heaving mystic metropolis. Tens of thousands of pilgrims from all over India were milling around the streets, unrolling their bedding on the pavements, cooking their breakfasts on portable stoves and haggling for provisions.
Inside the shrine enclosure, ecstatics and madmen crammed into a succession of Mughal mosques, tombs and pavilions. The entire complex was alive with the intoxicating smell of the roses that the devotees carried in sweet-smelling punnets to pour on the saint's grave. The numbers were amazing, but what was perhaps a bigger miracle was the number of different traditions from which the pilgrims were drawn. Many were Muslim, but there were also huge numbers of Hindus, as well as the odd Sikh and Christian, all queuing to pay their respects to the saint.
By 1998, I had been a journalist in India on and off for a decade, and much of my time had been spent covering violence between Hindus and Muslims. My very first report in 1989 had been from the disputed holy site of Ayodhya - whose Babri Mosque was finally destroyed by a Hindu mob on 6 December 1992 - and since then the growing communal tensions had consistently been one of the most depressing, and frightening, aspects of modern India. In a land increasingly polarised into rival religious groups, I found the gathering of different communities at Ajmer all the more remarkable. Here for once, you saw religion acting to bring people together, not to divide them. Sufism was not just something ethereal and otherworldly, so much as a powerful religious force that demonstrably acted as a balm on India's festering religious wounds.
I asked one group of Hindu pilgrims if they were made to feel welcome at a Muslim shrine. "Of course," replied their leader, a trader from Bhuj. "We never have any trouble in these Sufi dargahs [shrines]. All gods are the same."
"There is one God only," agreed his wife. "We are friends with our Muslim brothers and we have faith in their pir [holy man]."
"And the Muslims do not mind you coming here?"
"Why would they mind?" said the pair, baffled by my question.
"Garib Nawaz is here," said the trader. "He looks after every one of his followers. No one goes away empty-handed."
You might imagine, from the way the pilgrims talk about Garib Nawaz, that they were coming to see a living holy man. But Garib Nawaz lived in the 13th century. All existence and all religions were one, maintained the saint, merely different manifestations of the same divine reality. What was important was not the empty external ritual of the mosque or temple, but simply to understand that divinity can best be reached through the gateway of the human heart - that we all have paradise within us.
Garib Nawaz, like many Sufis, believed strongly in the power of music and poetry to move devotees towards greater love of God. Meditation, music, song and dance were seen as helping devotees to focus their whole being on the divine, and to do this so intensely that the soul was both destroyed and resurrected.
Today in Sufi shrines, you can still see the devotees being lifted by music into a state of spiritual ecstasy. Moreover, many Sufis believe that music is not only a path to the divine, but a means of spiritual healing: if there are diseases that appear to be physical, but have their root in an affliction of the spirit, these can be cured by listening to qawwali, the mesmerising love song of the Indian Sufi. In India, the lyrics of qawwali songs have always been sung not in the court Persian of the Muslim immigrants, but in the local Hindi vernacular used by the ordinary people, and they draw on symbols taken from dusty roads and running water, the dried-up thorn bush and the blessings of rain - images that speak directly and forcefully to simple folk of either religion.
Yet just as this use of music has brought the Sufis closer to their Hindu neighbours, so it has also sometimes acted to divide them from their Muslim brothers. Throughout Islamic history, there have been puritanical reform movements which have spoken out against the Sufis and their poetry and music, claiming that Sufi practices are infections from Christianity and Hinduism, quite alien to the original principles of Islam. According to Najaf Haider, professor of medieval history at Jawaharlal Nehru University in Delhi, to whom I talked about this, conflicts between the Sufis and orthodox Muslims of a more Koranic bent have always been inevitable, for the two have fundamentally different conceptions of their relationship with God:
"In orthodox Islam, the object of creation is the worship of God," said Najaf. "This is a relationship of subordination - a one-way relationship in which God is the master and the devotee is the slave. This relationship means that if you worship God, you will get rewarded - on the Day of Judgement, you will go to paradise - and if you do not, you go to hell. The Sufis completely rejected this idea and argued that God should be worshipped not because He has commanded us to, but because He's such a loveable being. So the cornerstone of Sufi ideology becomes love. As a result, all traditions are tolerated because, in the opinion of the Sufis, anyone is capable of expressing his or her love for God, and that transcends religious associations, gender or your place in the social order. That's the reason why Sufis became so popular and that's also the reason why they were anathematised by the orthodox."
The most formidable of all the anti-Sufi movements were the Wahhabis of Arabia, the progenitors of modern Islamic fundamentalism who, on coming to power in the early 19th century, destroyed all the Sufi and Shia shrines in Arabia and Iraq, including the tombs of the descendants of the Prophet.
At the time, most Muslims regarded the Wahhabis as an alien sect existing in a state bordering on kufr (disbelief). However, since then, the Wahhabis have used their astonishing oil wealth - the Wahhabi Saudi kingdom still controls one-quarter of the world's petroleum reserves - to try to suppress Sufism and other tolerant forms of popular Islam, and to remake the entire religion in their own, puritanical image. The Saudis now dominate as much as 95 per cent of all Arabic-language newspapers, magazines and radio and television stations. Many contemporary Muslims have been taught a story of Islamic religious tradition from which Sufism is rigorously excluded.
I first came across strongly anti-Sufi sentiments last autumn, when I visited a shrine just outside Peshawar in the North-West Frontier Province of Pakistan. The Sufi shrine of Rahman Baba has for centuries been a place where Muslim musicians and poets have gathered. The shrine is built around the tomb of a mystic poet whose Sufi verses in the Pashtun language have led to him being described as the "Nightingale of Peshawar". A friend who used to live nearby during the 1980s advised me to go to the shrine on Thursday nights, as that was the time when crowds of Afghan refugee musicians would go and sing songs to their saint by the light of the moon - a sight he described as unforgettable.
Since my friend left Peshawar, however, much has changed. Two Saudi-funded madrasas (religious schools) have been built on the road leading to the shrine. They have sought to halt what they see as the un-Islamic practices of the shrine.
"My family has been singing here for generations," I was told by Tila Mohammed, one of the guardians. "But now these Arabs come here and create trouble. They tell us that what we do is wrong. They ask people who are singing to stop. Sometimes arguments break out - even fights. Before the Afghan war, there was nothing like this: the Pashtuns here love Sufism. But then the Saudis came, and now this happens more and more frequently. Our way is pacifist. We love. We never fight."
This anti-Sufi movement seems to be spreading to India, and is increasingly taking hold of the Muslims of Delhi, where I now live. The Sufi village of Nizamuddin has always been one of my favourite areas of the Indian capital. As you walk, clouds of charcoal smoke waft into the air, and with it the scent of grilling meat floats out over streets bustling with pilgrims, madrasa students, sellers of rose petals and little boys playing cricket.
On one side lies the destination to which the crowds of pilgrims are heading: a warren of ever-narrowing lanes and alleys leading towards the shrine of India's most revered Sufi saint, Hazrat Nizamuddin Auliya. Yet only a short distance from the shrine towers an institution that embodies a quite different face of Islam.
The Tablighi Merkaz is a modern grey concrete structure housing the world headquarters of an austere Islamic reformist movement called the Tablighi Jama'at. The Tablighis are now probably the largest Muslim missionary movement in the world - they have a large British following based in Dewsbury and Nottingham. Although they advocate a return to the fundamentals of the Koran, they are not Wahhabi, but their beliefs derive from similar theological traditions. They share the Wahhabis' suspicion of popular Sufi practices and they are slowly undermining Islam's most tolerant and syncretic incarnation, just when that face of Islam is most needed to heal the growing breach between Islam and other religions. Their effect on the shrine of Nizamuddin is the same. For the Tablighis strongly oppose the popular devotional rituals associated with Sufi shrines, saying that they encourage un-Koranic practices such as idolatry, music, dancing and the veneration of dead saints.
This was certainly the view of Amin, an aircraft engineer from Dacca, who was busy trying to persuade passing pilgrims to turn away from their destination. "We Tablighis don't like tomb worship," he said, politely but firmly. "We believe there is no power but the Almighty. I invite these people who come to Nizamuddin to turn away from their errors and return to the true path of the Koran. Do not pray to a corpse, I tell them: Nizamuddin is dead now. So go to the mosque, not to a grave, and there tell your problems to Almighty God. Superstition leads to Jahannam [hell], but the path of true Islam leads to Jannah [paradise]."
"What sort of paradise?" I asked.
"The paradise of the Koran, of course," said Amin.
"But what is it like?"
"It is beyond all human imagination," said Amin. "But according to our beliefs, there will be many levels of paradise, eight in all, with a place for each believer. There will be couches to lie in the shade, and rivers of milk and honey and spring water . . ."
"So," I asked, "what do you think of the Sufi idea that God can be found in the human heart?"
"Paradise within us?" said Amin, raising his eyebrows. "No, no: this is emotional talk - a dream only. Is there evidence for this in the Koran? Real Islam is more disciplined than that: there are rules and regulations that must be followed: how to eat, how to wash, even how to clip your moustache. There is nothing in the Koran about paradise within the body. It is outside. To get there, you must follow the commands of the Almighty. Then when you die, inshallah, that will be where your journey ends."
Here, it seemed to me, lay some sort of crux - a small but important clash of civilisations - not between east and west, but within Islam itself. Between the ways of the orthodox Tablighis and the customs of the heterodox Sufis lay not just two different understandings of Islam, but two entirely different conceptions of how to live, how to die, and how to make the final and most important and difficult journey of all, to paradise. The question now is: which of the two is likely to win? There is no doubt, throughout the Muslim world, that this anti-Sufi movement is growing in strength; but equally, judging by what you can see of the clear popularity of the shrine of Nizamuddin, the Sufi way is not going down without a fight.
Leaving Amin at the doors of his Tablighi headquarters, I headed on down into the alleys of Nizamuddin. Taking off my sandals at the entrance to the shrine, I got into conversation with Hussein, the old man who looks after the pilgrims' shoes. I asked what he thought of the Tablighis. "These people should not prevent anyone from coming," he said. "Why should they tell people not to pray to the saint? Look around you, at all the streams of people coming: everyone in Delhi knows about the power of Nizamuddin. Everyone knows that if your heart is pure and you ask him something, he cannot refuse you.
"I have felt his power in my own life: I lost my hut in a slum-clearance operation ten years ago. That was the first time I came here. I was hungry and I had nothing. But I prayed to the saint and through him I found a place to stay, and a way of supporting myself. I tell you: if anybody abuses Nizamuddin Auliya in front of me, I will be the first to defend him - with my knife if need be."
Ayesha, a veiled Muslim lady, had overheard Hussein and came forward: "There's no reason why the Tablighis should stop anyone from coming. Don't think the people of India will let the Tablighis take away their saint. Just see the crowds. This shrine will never be allowed to die."
It was a Thursday, the most important evening in the life of Sufi shrines, when through the qawwali music the spiritual life of the shrine reaches its climax. Huge crowds of pilgrims were sitting cross-legged in the forecourt, and the first of a long train of qawwali singers began to strike up. Around them was an incredible press of people: Muslim grandmothers from Bengal in black chadors, Punjabi Sikhs in their blue turbans, Hindu women from southern India with large red bindis, all coming to use Nizamuddin as the intermediary for their prayers.
Slowly, as the crowds thickened, the tempo rose, and some of the pilgrims began to sink into a state of trance. Old men were swaying from side to side, arms extended, hands cupped in supplication, lost to the world; women were tossing their hair from side to side; and the first of a succession of dervishes rose to his feet to dance. The atmosphere, already heavy with the rich scent of rose petals, grew thicker still, filled with the softly mouthed and murmured prayers, the passionate incantations and expectations of ten thousand pilgrims.
I left them there, with their different prayers, still seeking paradise in that most elusive of all destinations, the human heart.
William Dalrymple is the New Statesman's south Asia correspondent. His most recent book, the Wolfson Prizewinning White Mughals, is being adapted for the National Theatre and BBC Films by Christopher Hampton