King Crimson performing at the Free Rolling Stones concert in Hyde Park, July 1969

It’s early 1974, British band Henry Cow are in the studio recording its second album, Unrest. One track features a forty-foot tape loop. Another is based on the fibonacci sequence, a structural device borrowed from Karlheinz Stockhausen. The track is in 55/8 time.

Henry Cow’s ethos is based on creating music that they aren’t yet competent to play. Groups that only play music they can already play, guitarist Fred Frith tells the NME, are “strait-jacketing themselves from the start”. The resulting music is as experimental as rock gets: intricate, atonal, related far more closely to the worlds of avant-garde classical and jazz than any other British band of the era. Critics are mostly bemused, fans disinterested.

But is it prog?

“Some people,” writes Mike Barnes in his new, exhaustive survey of British prog, “don’t think Unrest should be called progressive rock because it sounds so unlike other progressive rock. But that is surely why it is absolutely quintessential progressive rock.”

And that right there is what is so charming and attractive about the genre, despite its many excesses and absurdities: it doesn’t really sound like anything else. In Barnes’ reading, it isn’t so much a genre as a category term for experimental rock music circa 1969–1976. What typified it was a collective desire among a generation of musicians to look beyond American models for their music, to find inspiration in more culturally honest and authentic – and yes, more serious – sources closer to home. Some looked to classical music both in its traditional forms and its more modernist and avant garde incarnations, some to folk music, some to free jazz, some to what we’d now call world music. All of them wanted to shake off the shackles of the three-minute pop song.

Barnes traces the beginnings of the movement back to psychedelia and the 60s underground scene, while acknowledging early progenitors like keyboard player Graham Bond. Bond brought together musicians from disparate backgrounds, among them Ginger Baker and Jack Bruce, and encouraged them to express their own musical identities – an unusual concept for the time. But it was in the underground where prog really began. It was there that different cultural forms were encouraged to bleed into one another – music, poetry, performance, and so on – and different musical traditions allowed to cohabit. And LSD led people to the revelatory experience of music as texture as much as melody and rhythm.

Rick Wakeman and the cast of King Arthur and the Knights of the Round table on ice, May 1975

In some respects, you might say prog’s demise was inherent in its beginnings. At its most banal, it could be an earnestly intellectual movement, not an emotional one. Pretty much every epoch in twentieth-century popular music was largely about big simple things: youth and sex and dancing to a 4/4 beat of primal goodness and depravity. Prog is the only big wave that yearned to be more grown up than it was. Other music might be about the thrill of being seventeen and exploding with life; prog often seemed more about the thrill of being fifty-three and a professor of music at Oxford.

Rather than take a chronological path through the period, Barnes wisely charts his way from band to band, scene to scene, with occasional digressions into sex and drugs and journalism, among other things. Although there are drawbacks to this approach, it does highlight one of the genres abiding strengths: the determined idiosyncrasy and individuality of the different artists. Any genre that encompasses both the ferocious, terrifying power of King Crimson circa Red and the swooning, surreal, jazz-inflected intimacy of Robert Wyatt’s solo work has got a lot going for it, and the author makes a compelling case for much of this most various of genres. He is a generous critic but also a discerning and judicious one, not afraid to call out bombast and self-indulgence when he sees it. He has the requisite respect for, but distance from, his subject and his account is neither too dry, too cynical, nor too in awe.

But Barnes also has a wider point to make. A New Day Yesterday isn’t just about the music, it is about a moment in time, a brief period when musicians were giddy with the possibilities of their music and accepted no limits on their ambitions – and when record labels were happy to invest in experimentation and fans in vast numbers willing to indulge it. On this reading, prog was swept away not just because it had run its course, but because what replaced it was more corporate-friendly: more marketable, more easily branded and more readily commoditised. Just another consumer good.

At 600-odd pages, this is likely to remain the definitive account of British prog. Nevertheless, it’s a shame there isn’t more exploration of prog’s wider European family. This is partly because so much of British prog did look to the complex musical heritage of the continent for inspiration, and it makes most sense in the context of a European tradition of music, but also because there was some extraordinary music being made. Barnes touches on bands like Focus, Can and Neu! But the likes of PFM and Magma – whose decision to sing in a language of their own invention makes other bands merely labouring away in 11/8 seem like footling lightweights – are missing entirely.

Whether or not you see prog as the very last gasp of the underground and the optimism that went with it, it’s hard not to feel sorry for them: all these wildly ambitious young people, who found themselves, almost overnight, washed up by the tides of history and fashion, beached and helpless. “God’s joke is preparing you for a world that won’t exist by the time you get there,” said Jakko Jakszyk – now in King Crimson – who turned twenty in 1978. “Everything I’d known as currency turned to dust.”

For all its reification of technique over feeling, its creative and other excesses – exemplified by Keith Emerson, gussied up like some space-age Liberace, playing a grand piano upside-down thirty-feet in the air – prog left a lot of intense and intensely beautiful music behind. And looking back at the prog generation from this distance, nothing comes to mind so much as that line from Auden’s eulogy for Yeats: “You were silly like us; your gift survived it all.”

A New Day Yesterday: UK Progressive Rock & The 1970s by Mike Barnes is published by Omnibus Books