If the celebrations across the UK are anything to judge by, Stephen Sondheim's reputation as the giant of musical theatre is indisputable. All year long, audiences have been throwing themselves at his feet in gratitude for half a century of work. From the BBC Prom this summer in honour of his 80th birthday to glitzy events at the Cheltenham Festival, the Donmar Warehouse and the Royal Festival Hall, the adoration has been unstinting.
Sondheim's relationship with the public was not always this straightforward. Avant-garde yet populist, fearsomely intellectual but bawdy, acerbic and yet in thrall to old-fashioned ideas of love, he personified many contradictions that made hard graft of his becoming an international treasure. Several of his biggest hits were initially flops. It took time for his dissonant, complex music and sparklingly urbane lyrics to work their way into audiences' hearts and minds. And yet, for more than 50 years since he first got the job of lyricist to Leonard Bernstein on West Side Story, Sondheim has kept on experimenting, pushing the boundaries of what a musical could be while still putting bums on seats everywhere from the Royal Opera House to Broadway.
That the man is still so passionate about his art form and has decided to share his erudition and his enthusiasm for it will delight his many fans. He is putting together a collection of the complete lyrics, and a hefty task it is: volume one, Finishing the Hat, runs to more than 400 pages plus appendices, yet it takes us up to the early 1980s only. In it, we learn the master's views on fashioning a good lyric, the secrets of theatrical success, and the pet hates and loves built up over a long and distinguished career.
For a writer as gifted as Sondheim, who adores the plasticity of language, here is an opportunity to set down in print what he has carefully crafted, but which usually passes by in a moment in the darkness of a theatre. Along with the song texts for all of his musicals from 1954-81 - including West Side Story, Company, Follies, A Little Night Music and Sweeney Todd - Sondheim treats us to the numbers that didn't make it to the stage. Given time and space to breathe on the page, the lyrics seem fresher and more ingenious than ever, darker-hued than you might remember them, constructed with eminent facility. And everywhere are the flights of verbal playfulness and skill for which he is famed: "In the depths of her interior/Were fears she was inferior/And something even eerier/But no one dared to query her/Superior exterior." Crafting a good line has never seemed so much fun as in Sondheim.
Along with the song texts appear the "comments, principles, heresies, grudges, whines and anecdotes" of the book's subtitle. Despite his evident warmth and high regard for Oscar Hammerstein - who was a stand-in father to Sondheim when, as a boy, he moved to Pennsylvania, and who first taught him how to write a lyric - the student has no qualms about turning his critical sights back on his mentor. Hammerstein may have been a pioneering musical playwright, but Sondheim finds his characterisation empty, his philosophy rose-tinted, his poetry too pretty. He reserves particular scorn for the line from The Sound of Music "Like a lark who is learning to pray", wondering how and why a lark might do so.
Other "heresies" include getting the knives out for Noël Coward and W S Gilbert. Sondheim is gracious and measured enough, however, to interrogate his own work with the same unflinching gaze, and in several cases he finds himself wanting. Maria in West Side Story sings, "It's alarming how charming I feel", but with hindsight he questions whether a Puerto Rican girl with a brother who is a gang leader would ever speak like this.
Yet Sondheim reserves most of his bile for the new fad for what he calls "jukebox" musicals - overblown spectaculars featuring the back catalogue of, say, Abba or Queen, strung together with barely a plot; similarly, the "metamusicals", such as Spamalot, which cannibalise the form by subverting it, leaving an empty shell. With new musicals costing around £15m to stage, you can see Sondheim's point. Why take a risk on work like his, with its unconventional music and bitingly ironic lyrics, with no sweetener or happy ending? Far better to go for a known quantity.
The author says the aim of this book is to teach us about the craft of writing lyrics. Whether there is anyone out there to be taught is a moot point. Sondheim inherited the musical theatre of Rodgers and Hammerstein and fused it with the muscularity of literary innovators such as Arthur Miller and Tennessee Williams. Yet that was in an age when theatre mattered, when it buzzed and hummed with the energy of a generation of bright young things. Today, Sondheim seems like the last knight guarding the holy grail from the hordes of bounty-hunters and mercenaries. This book, then, is in part a dialogue with himself, a final hurrah from the master of his art, and also a reminder of what theatre meant in the past century but no longer means.
Finishing the Hat: Collected Lyrics (1954-81), With Attendant Comments, Principles, Heresies, Grudges, Whines and Anecdotes
Virgin Books, 480pp, £30
Suzy Klein is a presenter for BBC Radio 3.