In The Gulag Archipelago, Alexander Solzhenitsyn reminded us that, for the KGB, modern psychology made the infliction of medieval-style physical torture redundant. Psychological tortures, devised by doctors, were just as painful and effective. As Bisher al-Rawi, just released back to Britain after four years in Cuba, told me recently: "I think the psychological effect of this experience, in my opinion, far outweighs the physical. I think physical [effects] you can just overcome. Psychological [effects] you live with, all your life."

Torture, Solzhenitsyn made clear, was highly effective. Not for obtaining the truth, of course, but for terrorising a population and for obtaining the all-important confessions - confessions that legitimised an entire system of repression. In their efforts to harvest the maximum intelligence from the 800 or so prisoners of the camp, many endured both physical and psychological torments. The physical ranged from beatings to being strapped in stress positions and force-fed and the attentions of the ERF (emergency response force). The psychological stress came from solitary confinements, bombardment with strange music, cruel taunts, mortal threats and, above all, the uncertainty for many about where and why they were being held or what future they faced.

In Andy Worthington's The Guantanamo Files, the whole story of the Cuban camp emerges as a ghastly experiment in which the terrorist suspects became guinea pigs in a vast experiment of methods to crack the human soul. Until the 11 September 2001 attacks, few of these techniques had been refined; few among the US military had been trained in their use. What intelligence emerged from this programme? Did it actually succeed in helping combat the menace of al-Qaeda? The short answer is: we still do not know. Most confessions extracted from US prisoners remain classified documents.

The Guantanamo Files is a powerful, essential and long-overdue piece of research, providing the first real Who's Who of those held at the Cuban base. Though his heart sides with the prisoners, Worthington tries hard to be objective. Yet, as he admits, there is so much of his account that is based on one-sided claims, whether of the US authorities or of prisoners in the camp.

Looking for the truth of Guantanamo is certainly hard where information is so polluted by propaganda, abusive techniques and the sheer ignorance of many involved, as well as restrictions on what those involved can safely say in public. Much of what we hear comes either in the form of allegations based on US intelligence we cannot see or from prisoners' lawyers who, while courageous in their pro bono work, have a professional obligation not to reveal harmful information about their clients.

The Guantanamo Files provides a refreshing examination of the accounts of prisoners themselves, culled mainly from transcripts of their Combatant Status Review Tribunals held at the base and released after a Freedom of Information request.

The dice are loaded against prisoners. With even the slightest connection with the Taliban or Islamic militancy judged as proof of enemy status, it is hard for prisoners to tell the honest truth. That is why many prisoners' stories are far from convincing: one after another says he was in Afghanistan for charity work, to look for a wife, out of pure curiosity, or even, incredibly, for the fishing. A prisoner might confirm he was in the Tora Bora mountains with Osama Bin Laden - but only because he stumbled into bad company as he tried to flee for Pakistan.

But, while the book races from one account to the next, it is most powerful when it examines the accounts in detail and compares one with another. Only then does it become clear that, though many prisoners are obviously lying, a great many others are being honest. Of the hundreds swept to Guantanamo, described by the Pentagon as the "worst of the worst", few emerge as any kind of "big fish" in al-Qaeda.

Instead, we see clear evidence of a post-9/11 sweep where the US military were in effect duped into rounding up the small guys - both by warlords in Afghanistan, who pocketed large rewards for their co-operation, but also by in telligence officers in Pakistan, who picked up equally large suitcases of cash.

Rushing into battle with little preparation, US soldiers not only swept up Islamist militants, but also picked up proven members of missionary groups such as al-Tablighi (a group devoted to a strictly non-violent version of jihad) as well as junior workers for Muslim charities with a proven record of humanitarian work.

Adel Hamed, a Sudan ese worker in Pakistan for a Saudi charity, the World Assembly of Muslim Youth, was condemned at a tribunal because the charity "supports terrorist ideals and causes". A dissenting US military officer pointed out that such NGOs also had "numerous employees and volunteer workers working in legitimate humanitarian roles". And yet the chairman of the tribunal tellingly ruled that, nevertheless, the case passed the "low evidentiary hurdle" of such hearings.

Time and again, detainees face allegations from accusers whose identities are kept secret. When their identities are revealed, the intelligence and accusation frequently can be proven as falsehoods. Faced with one such wild anonymous allegation (that he ran a "network of ma drasas" able to field 5,000 soldiers for al-Qaeda), Abdul Salim Siddiqui, a Pakistani shopkeeper, told his tribunal: "The person who made these allegations was either drunk or didn't have a brain." After three years of Siddiqui trying to prove his innocence, a board member finally told him: "I believe you," and he was released.

Such examples as these provide evidence not only as to why those at Guantanamo deserve fair trials, but also to why so much secret intelligence cannot be judged reliable until fully tested by public scrutiny.