“Lifter Puller played a show in Albuquerque and there was a huge fight at the show. It was this club’s first night open, and it was crazy. It was like everyone in the club knew what side of the fight they were on, and they were all fighting, except us, and the cops came.”

Craig Finn, back home in Brooklyn on a short break from his US tour alongside Japandroids, is remembering one of the stranger nights with the scratchy, squealing Minnesota quartet he fronted in the declining years of the last century and at the very start of this one. “We got invited to this party once the fight was cleared up, and they’ve got these TV monitors up. When we got there, they were playing geriatric porn. Old people. And it’s strange. It’s a weird vibe. But we get some drinks, and we’re settling in, talking to a few people. At some point, people start cheering, and I look up, and now on these monitors it’s no longer geriatric people having sex, it’s an immense man, an obese man, and a normal sized woman. They’re cheering - I can’t work out why - and then I realise the obese guy is at the party.

“Near the end of the night I find myself in a conversation with said guy. I don’t know why, but I felt I needed to acknowledge his performance on screen. So I said, ‘Hey! I saw you up there. That was really great. Do you do a lot of that?’ He said, ‘Well, I’m trying to break in to it.’ We talked a little bit about it, and he said he mainly had to go out to California for it. Then - and I don’t know why I said this, other than to try to be conversational – I said, ‘Is there anything I might have seen you in?’ It just came out of my mouth. It wasn’t like something you might say if you met an actor. What he said in reply to that was the best. ‘I’m not sure. Usually I’m just the guy jacking off in the background.’”

It’s the kind of story that might have turned into one of Finn’s songs, with Lifter Puller, with the rock & roll band the Hold Steady, who gave him a career, or maybe even one of this three, rather more measured solo albums, the latest of which – We All Want the Same Things – comes out on March 24. Finn’s writing - hyperactive, dripping with internal rhymes and allusions both biblical and pop cultural, often telling the stories of individuals placed in situations just beyond the edge of their control - has been the linking thread in both bands and solo. He’s gone, though, from writing about the teenagers in the clubs, to the adults who have to negotiate the grey areas in life.

“I think this is the modern love album,” he says of the new record. “A lot of the songs are about people making partnerships or uneasy alliances to get through the world. I don’t want to say it’s a cynical view of love, because I think there’s beauty in making those partnerships and making teams, but it’s not the Disney version of love. If you are 28, you have that summer where you go to nine weddings. At my age - 45 - some of those are unravelling. But the thing is that if you talk to people you hear them say, ‘We’re such a good team - we get the kids to school on time.’ And so a lot of these songs concern two people. Some of them aren’t necessarily romantic, but they’re teams of people trying to complete something.”

The solo records return to the character-based writing Finn had largely eschewed on the last two Hold Steady albums, Heaven Is Whenever and Teeth Dreams, even if his characters now are often very different from the recurring figures who peopled his songs for so long. “With the Hold Steady, I felt I couldn’t continue on with the characters without it seeming like a sequel,” he says. “Crocodile Dundee 3 doesn’t have the power of the first one, and I say that having watched it within the last few years, and realising that it was made 13 years after even Crocodile Dundee 2. I guess I felt there are so many people in this new world that visiting them for a song at a time might be more interesting than putting them through a number of phases.”

Finn’s life was changed as a teenager, in the suburbs of Minneapolis, by hardcore punk. He’d loved the Bay City Rollers as a kid, then made his way haphazardly into music fandom (“I got my dad to take me to see Styx, and there was a point where my two favourite bands were Styx and the Ramones. Now that doesn’t make any sense to me, but at the time it was just the two things that I liked”). The John the Baptists for him were the Replacements, who sang about streets he knew, places he went. And then another kid at school suggested he try Black Flag. “I said, ‘No, I think that’s going to be too hard for me.’ He said, ‘Well, do you like the Replacements’ album Stink?’ And I said, ‘Yeah, that’s my favourite one.’ He said, ‘I’ll make you a tape.’ And then I realised I was already listening to something like hardcore.”

Finn has mythologised his teenage hardcore years in song repeatedly with the Hold Steady, but the “all ages hardcore matinee shows” of 'Massive Nights' made a profound impact. “The first year I went to them, they were terrifying, and that was part of the appeal. And then as I got older, I realised they were just people on the same journey I was on. They always smelled like clove cigarettes. You’d go in this little box and everyone would smoke clove cigarettes. I have a very olfactory memory of that. The shows would be from four o’clock to eight o’clock, and there would be three or four bands. My friends and I would get there at 3.30 because we did not want to miss - literally - one second. Now you go to see a band, and you time it so you get there right when they’re about to go on. But we wanted to see the first, second and third band. We wanted to see them tune up. We wanted to stand up close and see what kind of gear they had. Because it was all new. It was the only way we could get information: if we start a band, that’s the amp we should get, or the guitar. That’s how you pin a button on your jacket to make it look cool. One of the sad things about adulthood [is that you can’t recapture that feeling].”

At those Sunday shows - the “Sunday night dance party, summer 1988” of 'Chicago Looked Tired Last Night' - Finn realised the power of the scene, a recurrent theme in his writing, his onstage preaching, his speech. “It was this idea of showing up and knowing you’d see people. And you would make friends there – people who went to other schools. Maybe you didn’t even have their phone number. It was like, they went to other schools but they were friendly with you on Sunday. That was very cool to me, and being a part of the scene. I’d heard about a Minneapolis rock scene where cool guys went to bars and drank beers, and there was a pecking order. I found that very romantic, so I wanted to be a part of something.”

With Lifter Puller, Finn became part of a new Minneapolis scene, playing alongside bands who had little in common musically - the boisterous punk of Dillinger 4, the hip hop of Atmosphere - but who shared an existence in the demi-monde of the city, and he found his voice as a writer imagining the lives of the people who were fucked up and fucking up, who came to the shows and haunted the afterparties. They became preserved in song as Nightclub Dwight, Katrina, The Eyepatch Guy and more in Lifter Puller’s songs. After that band broke up and Finn moved to New York with no intention of making music, he ended up writing about a new group of characters - Charlemagne, Holly and Gideon - and the Hold Steady was born.

It’s February 2007 in east London, and the Hold Steady have finished their first headline show in the UK. I’ve left the venue and gone to a nearby pub, when their then publicist calls, asking if I want to meet Finn (I had been championing the band in the Guardian for several months). When I get back to the venue, I meet a dishevelled, tired and already really very drunk Finn, who gets only drunker over the next couple of hours, as we talk about sports, punk rock, more sports, and more punk rock, until he pulls out a hotel key card and asks if I have any idea where he’s meant to be staying that night. I do; it’s on my way home, and I drop him off. I’m slightly surprised he makes it from the cab to the lobby

That was the start of a friendship between us that has lasted a decade: my kids were a little taken aback at the contrast between the easygoing, slightly drawling man who comes to our house when he’s in town, and the berserk, hyperactive caricature they saw on stage, the guy you don’t want to get too close to in a bar, because however fascinating he seems, there’s the possibility he might get you dragged into something horrific.

That year, the Hold Steady went from being a bar band with a verbose frontman, who’d never left the US, to the brink of something. They drank, they played, they drank some more, they played some more. The reviews got more and more ecstatic, the crowds bigger and wilder, all responding to the band’s evident joy at their good fortune, and their sincere belief that playing rock & roll was just about the highest calling someone could have. “It seems like a blur, and it was crazy. But at the same time it was amazing, and really fun. But there was a part of me that didn’t quite trust it.”

The highs, though, were very high: supporting the Rolling Stones at Slane Castle, duetting with Bruce Springsteen on Rosalita at a charity gig in New York, after the Hold Steady had played Atlantic City. “When we finished, he was in the wings, and he said, ‘Good job, man.’ And then he called all artists up on stage, and said, ‘Who knows the words to Rosalita?’ He kind of looked at me. Funny enough, we’d talked about doing Rosalita over Atlantic City, so I’d learned the song pretty well. So we kick it off, and I sing the first verse, and I think Jesse Malin sings the second, and the third is Damon from Badly Drawn Boy. And then he kind of motions, like, take it home after the big instrumental break - so I got to do the whole “Your poppa says he knows that I don’t have any money’ section with him, on the mic, I remember thinking, ‘This is happening. Please, don’t think about anything else for the next five minutes.’”

The band’s recklesssness meant things could go badly wrong. I recall seeing guitarist Tad Kubler, very drunk onstage in Nottingham - “He was a leader. He was the first guy into battle and we all followed him” - attempting to play a solo while sliding down a bannister at the side of the stage at the Rescue Rooms. At the bottom, he fell to the floor, landed face first, got up and carried on soloing. The next day he discovered he had broken his wrist.

The fans followed suit. “One of the things is, we consciously encouraged a level of drinking at our shows that can sometimes come back to bite you,” Finn says. “There was one guy in Belfast that met me back behind the club. I’d seen him at the show, and he’d been really enjoying himself. But at this point he had gotten himself kind of angry. He wanted me to call his friend on his phone, and I didn’t really want to do that. And he grabbed me, hard. I ran away from him, and he chased me to the bus. It ended with him punching the bus while we were all locked on it. But he was singing Hold Steady songs. The bus driver chased him off with a stick. That guy came to the show really excited, and that’s how his evening ended.”

For nine of the ten years I’ve known Finn, he has looked, well, like an accountant. Clean shaven, trimmed hair, sober enough. That first year he looked a mess. Hair all over the place, bedraggled, bearded. “I wasn’t taking as much care of myself as I normally do, and I was also getting a divorce back home. And my mental state was kind of … just keep going. I was going out every night after the show because I was unrooted. I was in bad shape. I think if you are presented with as many opportunities as we were, to play all these shows, to keep coming back to England, to take all those overnight flights and hit the ground running and then play the show, and you’re just on this hamster wheel, here has to be a fatigue that sets in. And I think that makes a rock & roll band a very difficult thing to sustain without a breakthrough. And in some ways I think that’s kind of what happened. There was an exhaustion that set in, and poor communication, and real health things that came up with the Hold Steady.”

The breakthrough never came. The Hold Steady plateaued. There was never a radio hit, and the rooms stopped getting bigger. The band’s manager at the time of their rise had warned them it might happen - “He told us: ‘This is one thing, but having a career is another; especially over here, this can go away” - and when it became clear they were going to stick a “beloved cult” level, Finn was prepared. “I wanted it to get bigger and bigger and bigger, but what happened made sense.”

Then, in autumn 2008, Kubler was diagnosed with pancreatitis, a European tour had to be cancelled at the last minute, and everything changed. For one thing, Kubler had to stop drinking. That may not sound that big a deal, but it fundamentally altered the dynamic of the band: where songwriting had previously often been based on jamming over a few six-packs, now it became a matter of matching Finn’s words to Kubler’s riffs, and they simply stopped writing as many songs. Since then, the band have released just two albums in seven years, compared to four in their first four years.

Onstage, too, things were different. The drinking Hold Steady had been a sloppy rock & roll band, messy in all the right ways. Afterwards, they swapped the sloppiness for the force of a juggernaut - not better or worse, but different - and Kubler cut a much more subdued figure in stage. “There was a before and after feeling,” Finn says. “Even now there is a before and after feeling. There were challenges that I’m not sure we’ve ever totally figured out. How we handled it was: obviously we’re not going to be sliding down handrails and breaking our wrists onstage anymore. Why don’t we just try to tighten up and make it pro. And to some people that seemed less exciting. But we gained a lot of mental health, physical health, spiritual health.”

Finn these days understands the value of not being the guy who has to be blind drunk to give everyone a good time. He gives up alcohol every year for Lent - this new album contains several songs he began a couple of years ago, when he decided to write a song every day of Lent, though not, sadly, the one in which he imagines Whitesnake’s David Coverdale stranded in a cheap motel in Minneapolis after a snowstorm shuts down the airport, entitled 'Dark Days For A Cocksman'. These past couple of years, for the first time in a long time, he’s been playing support shows as a solo artist, with bands who’ve taken inspiration from him, and from the same things as him - not just Japandroids, but Titus Andronicus and Mariachi El Bronx. Without all the worries of his parent band - making payroll, making sure everyone has health insurance, helping run the business, which is what a career band is - he’s been able to be remarkably prolific, and to be able to say no to things he doesn’t like and yes to things he does, without the risk of pissing anyone else off. He’s enjoyed trading down from living “like a vampire” in the darkness of a tour bus to travelling in a van, where “if you want lunch, you can find somewhere that’s pleasing. And you’re lining yourself up with a more human schedule.”

And through it all, from it all, the songs keep coming; the vignettes of American lives somewhere off in the margins, just at the point where the shadows creep across the street and things start getting sketchy, no matter how much older Finn or his characters are getting.

We All Want the Same Things is released on Partisan on March 24. Craig Finn plays Ramsgate Music Hall on April 16, supported by Her’s, which is - coincidentally - the day he will start to drink again after Lent