Or, Promise and Compromise. Or, Anticipation and Regret. Or, Fulfilment and Frustration. Any of these would work as an alternative title to this two-CD, two-and-a-half-hour compilation, subtitled '30 years in the wilderness with Stephen Duffy and the Lilac Time' and often feeling like exactly that, even without the film and book which complete this mammoth and rather unexpected feat of TinTinalia. It's an idiosyncratic anthology, selected by Stephen himself and with accompanying sleevenotes in which he both self-mythologizes furiously and is courageously open about his own doubts and disappointments with the work presented. Largely steering clear of singles and (non-)hits, the collection is surprisingly cohesive and, in places, a thing of understated beauty. But there are moments too where the magic fades, and a smooth surface veneer stands in for any real emotional engagement or melodic inventiveness.

Stephen 'Tin Tin' Duffy first sashayed into the pop consciousness in the mid-1980s, a time when you could still get away with referencing Dorothy Parker and Maxfield Parrish in Smash Hits. With perfectly manicured eyebrows seemingly permanently raised, he launched debut album The Ups and Downs with an exhibition of paintings and photography alongside his brother Nick, with whom he simultaneously founded a design agency. In the post-New Romantic 80s, this kind of caper had been elevated to a popular art form, and Duffy seemed born to it: a Birmingham art school graduate, he'd been a founder member of Duran Duran (pre-Le Bon), and his two top twenty singles ('Kiss Me' and 'Icing on the Cake') were as archly disposable examples of New Pop confectionery as the Paul Morleys and Robert Elmses of the time could ever wish for. And yet, on the cover of The Ups and Downs he already appeared intensely uncomfortable: pouting for England, his tall slim figure is contorted into the shape almost of a question mark, arms self-consciously folded against his expensive and immaculate dark Italian suit.

What happened next was a sequence of events that would repeat themselves with depressing regularity throughout Stephen's career. The record company took exception to the running order and title of the follow-up album (changing it from Cocksure to Because We Love You), and when it took a dive Stephen was unceremoniously dropped with it. The album is represented here by the jazzy 'Sunday Supplement' and the plaintive 'Julie Christie' — both of them gauche and gorgeous, hopelessly romantic and a million miles from the upbeat electropop that Virgin/10 obviously wanted.

That could have been where the story ended; in fact, it's where it really begins. Returning to Birmingham and finding solace in the then still unfashionable sounds of Nick Drake and The Incredible String Band, Stephen and Nick Duffy formed The Lilac Time and released their eponymous debut album on Midlands indie Swordfish in 1987. 'Black Velvet' captures their early appeal perfectly: a droning melody that makes the most of Stephen's rather limited, reedy vocals, an accordion cutting through the gently-picked acoustic guitar like a breeze through the birch grove, and a dry, melancholy lyric that makes the first of many references to snowbound streets as an all-purpose metaphor for loneliness and romance.

The album was picked up and polished by Fontana the following year, and gained some TV and radio play (a video on ITV's Saturday morning The Chart Show here, a Janice Long session there), but failed to give Stock, Aitken and Waterman too many sleepless nights. A further two LPs were compromised by record company interference, but 'The Lost Girl in the Midnight Sun' from 1989's Paradise Circus is a standout, its driving, dreamy progress recalling Tomorrow's 'My White Bicycle' as it builds into a call to arms for all the dreamers and romantic malcontents alienated by the Thatcherite decade then drawing to a close. Though ignored at the time, it would resonate throughout the years to come.

Alas, fine songs were relegated to B-sides (three are exhumed here), and 1990's And Love for All came out sounding bland and uninspired; the band were dropped again, before Creation took them on for 1991's Astronauts, a marked improvement with its stripped-back, bare bones sound, represented here by the smudged, autumnal 'Grey Skies and Work Things' and the elegiac 'Madresfield.' Inspired by the same stately pile as Waugh's Brideshead Revisited, this song's melancholy depiction of an abandoned, wintry cricket pavilion is perhaps a metaphor for the band at the time, as a despondent Duffy would abandon recording the album partway through and leave The Lilac Time to 'sleep beneath the snow' — though not, as it turned out, for good.

Stephen went on to record 1993's Music in Colors (American spelling deliberate) with classical music's answer to Jamie Oliver, Nigel Kennedy, and still claims it as one of his favourites, though on the strength of the three tracks presented here it's hard to see why — only 'A Fall from the Sky' emerges unscathed by Kennedy's caterwauling fiddle. When that album was roundly ignored, Stephen headed to the states to make 1995's Duffy with producer Mitch Easter and US power pop janglers the Velvet Crush, a record he considered releasing under the name of Harrison Browne, which tells you all you really need to know about its canon-fed soft rock classicism. Strangely, this was regarded as Stephen's Britpop album, and to be fair the two tracks included hereon — 'A Child is Waiting' and 'Ghetto Child' — are enjoyable pastiches the equal of anything that, say, Teenage Fanclub were doing at the time.

The album flopped, of course, though Stephen did make a return to the UK charts with the one-off single 'Hanging Around' — not included here — as part of MeMeMe with the ubiquitous Alex James and Justin Welsh from Elastica. This persuaded RCA to let him have another stab, and although 1998's I Love My Friends fared no better than its predecessor, songs such as the simple, Ray Davies-ish 'The Postcard' find Stephen at the top of his game, recording an affecting tale of death and teenage romance with just voice, acoustic guitar and a smear of harmonica. In the sleevenotes he reveals that he was initially embarrassed by the directness of the lyric, further reinforcing my suspicion that the man has never been the best judge of his own work. 'The Deal' is perhaps even better, a deceptively jaunty cousin of Leonard Cohen's 'Stranger Song' with more than a touch of Nick Drake's dark English wistfulness.

Not that anyone cared. Finding himself once more without a deal, Duffy was bailed out by royalties from the three songs he'd co-written on Stunt, the multi-million-selling breakthrough album by Canada's irritatingly whacky Barenaked Ladies. Seven years later he would repeat this trick with more deliberate intent and even greater reward, co-writing and co-producing Robbie Williams' Intensive Care album, and even heading up his touring band. This steady flow of filthy lucre would subsidise the reformation of the Lilac Time, who from being forgotten also-rans in the late eighties had gradually built up a cult following that included the organisers of the Green Man Festival, who booked the band for a valedictory comeback performance in 2007. Since 1999 The Lilac Time have recorded a further four albums, selections from which make up most of the second disc of this compilation. There's a polished maturity to this later work that recalls Bookends-period Simon and Garfunkel, rather than the Donovan/Drake-isms of the band's earlier incarnation, with songs about growing old, love and loss coloured by steel guitar and comfortable harmonies. The album ends with a new version of 'Kiss Me,' reworked as a harmonium-led drone that is probably more of a perverse gesture than an indication of things to come, but for me its suggestion of Six Organs of Admittance-styled experimentation makes it more appealing than most of the dozen songs that precede it.

Ultimately though, despite his weaknesses, I would still rather listen to Stephen Duffy than almost any more celebrated contemporary singer-songwriter. His songs never (or very rarely) pretend to be anything other than what they are: flawed products of middle England, in debt to their forebears yet charmingly understated, dealing with depression, heartbreak and failure without self-pity, and with moments of joy and beauty without any unnecessary whooping or flag-waving. At best, they convey all the romance and melancholy of walking home along an empty suburban street at twilight, half-drunk and alone but with your life stretched out behind you, almost as though it were an exquisite pattern leading up to just that moment. At worst, they have all the banality and emptiness of the morning after. Somewhere between the two: that's the Lilac Time.