On the evening of 20 October this year a memorial service was held at Amherst College, Massachusetts, for the writer David Foster Wallace. Wallace, who graduated from Amherst in 1985, had hanged himself the previous month at his home in California. He was 46.

Since the publication of his voluminous and extravagantly ambitious second novel, Infinite Jest, Wallace had been widely acknowledged as the finest American writer of his generation. When Infinite Jest came out, in 1996, New York magazine's Walter Kirn declared that the competition had been "obliterated". It was, he wrote, "as though Paul Bunyan had joined the NFL, or Wittgenstein had gone on Jeopardy!". More or less instantly, Infinite Jest became the benchmark against which any ambitious and intellectually curious young novelist would measure himself. And so, when Wallace died, American fiction lost its lodestar. His suicide was a kind of Kurt Cobain moment, a profound generational shock. (Cobain, frontman of the rock band Nirvana, shot himself at home in Seattle in 1994. He was 27.)

Wallace had begun to write fiction while at Amherst, and his first novel, the antic and exuberant The Broom of the System, published when he was 24, was based on his senior English thesis. One of his professors, Andrew Parker, told the congregation at the Amherst service at Johnson Chapel how Wallace had insisted on writing a thesis in the philosophy department at the same time, and had still managed to graduate summa cum laude in both subjects. Sue Dickman, an Amherst alumna, remembered the year Wallace spent there as an instructor after an abortive attempt to start a PhD at Harvard; she recalled how he would come to class with a tennis racket and let his students take breaks so he could smoke.

The mourners also heard from Mark Costello, Wallace's college room-mate in the early 1980s, who implored them not to forget "how painful Dave's day-to-day life was". Wallace had suffered from depression since adolescence, and was hospitalised during his sophomore year at Amherst. Later, he briefly shared an apartment with Costello in Somerville, near Boston. Wallace was drinking heavily and experimenting enthusiastically with drugs, and eventually ended up in McLean Hospital, a psychiatric institution that had previously counted the poets Sylvia Plath and Robert Lowell among its patients. The "power of death", Costello said, was his constant companion, and "eventually it cornered him and killed him".

While he was at McLean, Wallace was prescribed Nardil, a powerful antidepressant that he would take for most of the next 20 years. By the summer of 2007, however, the drug had begun to have unpleasant side effects, and it was decided that he would come off it. Doing so had catastrophic results. "Severe depression came back," Wallace's father, James, told the online magazine Salon. "They tried all kinds of things. He was hospitalised twice . . . and had a series of electroconvulsive therapy treatments, which just really left him very shaky and very fragile and unable to sleep."

By this summer, Wallace hadn't written anything for more than a year. He had been granted medical leave from his teaching job at Pomona College in Claremont, California, and had stopped going to the gym with his friend and colleague John Seery, where they had been the only "skinny-ass egghead types" in the place.

When, in mid-August, his wife, Karen Green, had to go away on family business, Wallace's parents came to stay with him in Claremont. "He was very emotional," his mother, Sally, said. "He was just terrified of so much. We would just try to hold him. He did tell me that he was glad I was his mom." James and Sally Wallace flew back home to Illinois at the end of the month. A fortnight later, Karen came home from a shopping trip to find her husband dead.

News of Wallace's suicide first broke on the Twitter page of the Brooklyn-based writer Edward Champion. By the time the mainstream media picked up the story, Champion was already gathering tributes and testimonials from the "literary community" on his blog, the comments box of which was soon filling up with expressions of shock and anguish from Wallace's fans. The confession of one devotee that "I haven't really cried over the death of someone I haven't met since Kurt Cobain" was typical.

Within a couple of days, Dave Eggers's McSweeney's ope ration had a virtual book of condolence up on its website, and people were posting reminiscences of meeting "DFW" at readings in out-of-the-way branches of Barnes & Noble, of taking writing classes with him and, in one case, of playing tennis against him in fifth grade. (Wallace had been an adolescent tennis prodigy, playing the junior circuit in the Midwest in the mid-Seventies. The first piece of extended narrative journalism he wrote, for Harper's magazine, was about summers spent "being driven through dawns to tournaments all over Illinois, Indiana and Iowa"; and Infinite Jest is, among other things, probably the finest, and certainly the longest, work of fiction about tennis ever written.)

The internet is a uniquely effective mechanism for processing public grief, of course, but there was something striking nevertheless about the sheer copiousness of the reaction to Wallace's death. This was partly because he was the kind of writer who attracts acolytes as well as admirers. The esoteric, self-enclosed quality of much of Wallace's fiction, not least the great thousand-odd-page slab of Infinite Jest itself, tended to inspire unusually fierce allegiance in many of his fans - the sort of obsessive who can tell you the likely effects of putting your head in a microwave (this being how the father of one of the central characters kills himself) because they've checked with a "med-school acquaintance", and who posts elaborate theories about arcane plot points online ("It sounds far-fetched, but check this out . . .").

Just as striking, and much more significant, however, was the reaction of other writers, Wallace's peers. There was spontaneous, unanimous agreement that he had been the best of them. Jonathan Franzen described Wallace as a "huge talent, our strongest rhetorical writer". In Zadie Smith's view, he had no "equal among living writers. He was an actual genius". He was also, everyone agreed, an unusually generous man: as "sweet a person" as Franzen had ever known, "meticulous about putting people at their ease", according to Smith.

Eggers posted a series of anecdotes about Wallace on the McSweeney's website. In one, he recalled the first time they met, in the mid-Nineties, at a diner in New York. While Wallace chewed tobacco, using a teacup in his lap as a spittoon (a habit he would never give up), he and Eggers "talked about how he'd grown up in Champaign-Urbana, Illinois, and how I'd gone to college there, how his father taught there, about the pleasures and quirks of east-central Illinois. There's something very strange and uniquely powerful about meeting a guy whose writing you find world-changing but who also comes from your part of the world."

Who else could have written a story about a marketing executive who plans to inject deadly cultures into a snack he’s testing with focus groups?

Wallace was, by his own description, an “infantile transplant” to Illinois from upstate New York: his father had done his PhD at Cornell University and had come west to teach philosophy at the University of Illinois in Champaign. His mother taught English at a local community college. Wallace described the atmosphere of the family home in an essay he wrote for Harper's: “Suppers often involved a game: if one of us children [Wallace had a sister, Amy] made a usage error, Mom would pretend to have a coughing fit that would go on and on until the relevant child had identified the relevant error and corrected it. It was all very self-ironic and light-hearted; but still, looking back, it seems a bit excessive to pretend that your small child is actually denying you oxygen by speaking incorrectly.”

Being raised as an "extreme usage fanatic" by a teacher of composition certainly left its mark on the mature Wallace's prose style. In another of his McSweeney's posts, Eggers remembered the time Wallace sent him a story, "Mister Squishy", together with a note asking that if it was to run in his magazine, it should do so under the pseudonym "Elizabeth Klemm". Eggers agreed ("we were so proud to publish it"), but the subterfuge didn't last long: Wallace was simply "too recognisable to hide, too singular to fool anyone". Who else could have written a 60-page story about a marketing executive who hatches a plan to inject deadly cultures into the "snack cakes" he is testing with a focus group? And all this in sentences of improbable length and fiendish complexity.

Take the passage in which the protagonist fantasises, while briefing his focus group, about how much ricin or botulinus it would take to bring the entire corporate snack industry to its knees. As he is doing so, he is struck by his ability to entertain such fantasies in "total subjective private" and then by the thought that half the people in the room with him are doing the same:

. . . Schmidt had a quick vision of them all in the conference room as like icebergs and/or floes, only the sharp caps showing, unknowing and -knowable to one another, and he imagined that it was only in marriage (and a good marriage, not the decorous dance of loneliness he'd watched his mother and father do for seventeen years but rather true conjugal intimacy) that partners allowed each other to see below the berg's cap's public mask and consented to be truly known, maybe even to the extent of not only letting the partner see the repulsive nest of moles under their left arm or the way after any sort of cold or viral infection the toenails on both feet turned a weird deep yellow for several weeks but even perhaps every once in a while sobbing in each other's arms late at night and pouring out the most ghastly private fears and thoughts of failure and impotence and terrible and thoroughgoing smallness . . .

This is writing of extraordinary syntactic control, and it is characteristic of what Eggers describes as Wallace's "dense, discursive, and insanely detailed style". The sentence continues for almost another page; the paragraph in which it occurs runs over four pages. Eggers says that he asked Wallace to consider breaking up some of the paragraphs before the story was published: "It was as if he were visiting the notion . . . for the first time. He was that kind of genius, whose understanding of the workings of his own fiction was, I think, largely separate from ideas of audience."

Actually, Eggers is missing something here, because Wallace thought more deeply about questions of audience and address than any other American writer of his generation. All that "insane" detail, and the formal strategies he borrowed from postmodernism (authorial intrusions, digressions, footnotes, flash-cutting between scenes and so on), were meant in fact to serve the rather traditional end of saying something about "what it is to be a fucking human being".

Wallace believed that each of us is "sort of marooned" inside our own skull, and that it is fiction's job to "aggravate this sense of entrapment and loneliness and death in people". It was the estranging apparatus of his style - the postmodern rhetorical devices, the hyperextended sentences - that was meant to do the aggravating.

That, at least, was the theory. However, Wallace was tormented by the thought that the “antagonistic elements” in his fiction might in fact just be manifestations of a pathological exhibitionism. He deplored his own “grossly sentimental affection for gags” and weakness for “formal stunt-pilotry” that served no narrative purpose. But he also understood that this predicament was not his exclusively; it was that of an entire generation of writers who, in a sense, had come too late – who had arrived, that is, just as the bold innovations of postmodern novelists such as John Barth, William Gaddis and Thomas Pynchon were being absorbed and neutered by the culture.

Wallace believed that each of us is “sort of marooned” inside our own skull and that it is fiction’s job to “aggravate this sense of entrapment”

This highly developed generational self-consciousness is one reason Wallace was held in such esteem by his peers: he held up a mirror to their own anxieties, and articulated them more clearly and honestly than they ever dared.

Infinite Jest bears the scars of Wallace's parricidal struggle with his influences. He insisted that he had wanted it to be an "extraordinarily sad" book about loneliness and addiction, rather than a postmodern one. And to this end, the novel's two principal settings are a tennis academy, which Wallace depicts as a kind of laboratory of obsession (he writes with considerable feeling about the psychic costs exacted by endless early-morning tennis drills), and a halfway house for recovering drug addicts and alcoholics. However, it is also crammed with set-ups and gags that could have come straight out of the fiction of Thomas Pynchon: a geopolitical agglomeration with the acronym ONAN; a gang of wheelchair-bound Québécois separatists; and a film so entertaining that it paralyses anyone who watches it.

At times it seems as if the novel is conducting an argument with itself - for instance, in a long scene in which Don Gately, a former drug addict who is now a live-in staffer at the half way house, goes to an Alcoholics Anonymous meeting in Boston. One of the residents in Gately's care is there, too, and complains about the "psychobabbly dialect" that's de rigueur at events like this. Gately admits that the "seminal little mini-epiphanies" routinely experienced by new inductees into AA come embalmed in language of "polyesterish" banality. Then someone else says they also find the sentimental argot hard to stomach - especially the habit the speakers have of saying they are "here but for the grace of God", which phrase, she points out, is "literally senseless", and should be used only when introducing a conditional clause. Wallace is flattering his hip and savvy readers here, inviting them to identify with this sophisticated cynicism. But it is also clear that we are meant at the same time to find something ridiculous and overwrought about someone who is driven to want to "put her head in a Radarange" by a home-spun solecism or two. Indeed, Wallace said later that the scene was designed to get his readers - privileged, educated Americans, most of them - to "confront stuff about spirituality and values", stuff "our generation needs to feel".

Infinite Jest turned out to be Wallace's last novel. He didn't stop writing fiction - there were two further collections of stories, the second of which, Oblivion, contains some of his finest work - but, as Mark Costello observed in his Amherst encomium, something changed: "If you sit down and read his prose from the early Nineties to later, you'll hear the music changing. You'll hear sentences getting longer and longer, with these wonderfully balanced dependent clauses."

According to the novelist and journalist Tom Bissell, who knew Wallace, this was also a shift in "world-view". Bissell told me: "I think the man who wrote Oblivion would not have been satisfied with the cross-dressing leader of a Québécois separatist group whose primary mode of transport is wheelchairs, or with the political acronym ONAN, both of which are kind of silly in the way Pynchon is silly, but not in the way the world ever feels silly. I believe he escaped the anxiety of his influence."

Wallace's deepening ambivalence about the moral as well as aesthetic legacy of postmodernism is especially noticeable in the non-fiction he wrote in the last decade of his life. In many of his essays, whether the brief was to write about Caribbean cruises or the Maine Lobster Festival, he can be seen grappling obsessively with all that "stuff about spirituality and values".

A good example is a piece he wrote after Harper's sent him back to Illinois to attend the state fair, and to gorge on corn dogs while watching the rural Midwest at play. "Getting Away From Already Pretty Much Being Away From It All" derives all its considerable force from the tension between Wallace's self-acknowledged "East Coast cynicism" and his yearning for a kind of authenticity.

He describes coming across a small hillock that, for some reason, has been covered in artificial grass: "a quick look under the edge of the fake-grass mat reveals the real grass underneath, flattened and already yellowing". Now, the postmodernist debunker in him would have been content to leave that image to stand for the emptiness and shoddiness of the state fair as a whole. But Wallace doesn't settle for simply unmasking the event as a sham; and this is partly because, for all his protestations that he is no longer "spiritually Midwestern", he remains of the place he is describing.

He understands that to be Midwestern is to be "marooned in a space whose emptiness is both physical and spiritual", and that what the state fair provides is a kind of temporary communal respite from that condition. As Bissell, himself from the Midwest, puts it, "in terms of literary persona, [Wallace] was temperamentally speaking a rural Midwesterner, intellectually speaking a high-wire postmodernist, and emotionally speaking an artist-as-priest type. I'm not sure anyone else has really managed to combine those qualities."

Wallace’s best essays are artefacts of this multifarious literary personality, dramatisations of its internal conflicts. Nowhere is this drama more powerfully performed than in the 15,000-word article he wrote for Rolling Stone about a week he spent following John McCain during his Republican primary battle, in 2000, with George W Bush. (The article was turned into a short book this year to coincide with McCain’s second run at the presidency.)

It is unsettling reading the piece now - after a Republican campaign this year conducted mostly from the sewer - to recall that eight years ago McCain represented to many the last, best hope for decency and truth in American politics. Even the most jaded hacks aboard McCain's bus, the Straight Talk Express, were wondering if "humanity and politics, shrewdness and decency" really could coexist. And Wallace finds himself as inspired as anyone: "It's difficult not to feel enthused and to really like this man and want to support him in just about any sort of feasible way you can think of."

Underpinning McCain's promise in 2000, of course, was "something riveting and unspinnable and true": that the candidate had been imprisoned and tortured during the Vietnam War, and spent five years in a box-sized cell at the "Hanoi Hilton". But whatever it was in his character that sustained McCain during those five years was shut away in that box, too. This, for Wallace, was the essential "paradox" of his campaign: the fact that the "box that makes McCain 'real' is, by definition, locked. Impenetrable." Which meant that it was up to the voter to decide whether McCain was "truly 'for real'".

What the essay is really about, therefore, is the "interior war" inside Wallace's own head between the need to believe in something larger than himself and his anxiety that the need to believe might be a sham or a fraud. And "fraud", Mark Costello reminded the mourners at Amherst, "was one of the worst words in his personal vocabulary".

In the end, Wallace took his own life. Perhaps the struggle to believe in something was too great; he had suffered too much. He was the ultimate victim of his own interior war.

Jonathan Derbyshire is a writer and philosopher