It came to Tom Waits in a terrible dream. In amongst the old clothes and shoes in the Salvation Army store was a stack of old vinyl albums, and as he flipped through them he chanced upon one of his own efforts. The sleeve stared at him almost reproachfully, and he knew something had to change. He didn't want to be a has-been, a '70s bargain-bin relic. He had to create something unique, "something you'd want to keep."

Like the most uncompromising singer-songwriters of the decade just past – Dylan, Young, Bowie, and others – Waits decided it was time to reinvent himself, to shake off everything he'd stood for. It was make or break, a bloodless coup that would dispense with all and any safety nets: manager Herb Cohen, producer Bones Howe, best friend Chuck E. Weiss and the gang of friends who'd frequented the Tropicana motel and Troubadour club; maybe even his label Elektra-Asylum. The baby might get thrown out with the bathwater, but it was that or rest on what scant laurels he could claim.

Having Kathleen Brennan – the young Irish-American girl he'd married in 1980 – at his side was the only green light Waits needed. Thinking back to his first albums, he realized he'd lived his life in reverse, starting out as a cautious old man before entering a radical middle age. By the time he was physically old, he figured, he might regain the innocence and intuitiveness of childhood. "I hatched out of the egg I was living in," he said, looking back on this time. "I'd nailed one foot to the floor and kept going in circles, making the same record."

Those who love Waits' '70s albums will take issue with this statement, yet the fact remains that up to this point Waits had borrowed a bunch of styles and mannerisms – from music, movies, books – and jumbled together his voice and style. However great the songs were, to his ears they were derivative, unoriginal.

Waits has given a lot of credit to Kathleen for exposing him to sounds that lay outside the commercial pop-rock spectrum. "She was the one that started playing bizarre music," he claimed. "She said, 'You can take this and this and put all this together. There's a place where all these things overlap. Field recordings and Caruso and tribal music and Lithuanian language records and Leadbelly. You can put that in a pot.'" The process that had tentatively begun with the harder-edged R&B of Blue Valentine (1978) and Heartattack and Vine (1979), then taken a twist with the aberrations on the soundtrack for Francis Ford Coppola's One From the Heart ('Circus Girl' and 'You Can't Unring a Bell'), now paved the way for a music that fell completely outside Waits' old vocabulary.

Waits has even claimed he'd never really listened to Captain Beefheart before meeting Kathleen – that, despite being signed by Beefheart's manager Herb Cohen and supporting the Captain's mentor Frank Zappa on tour, he was oblivious to the gruff-voiced Dadaist genius born Don Van Vliet. Where the bruising 'Heartattack and Vine' hinted at Beefheart's influence, Waits' new songs made the debt clearer still.

Whether or not it was Kathleen who turned her husband on to Beefheart matters less than the fact that she was the catalyst for change. "It was certainly not the first time that a relationship with a strong and well-organized woman has had that kind of effect on an artist's life," says Mike Melvoin, keyboard player on The Heart of Saturday Night (1974) and Nighthawks at the Diner (1975). But Melvoin also believes Brennan "put a stake through the heart of various things" in order to free Waits from his past.

"I don't blame Tom for it, I blame Kathleen," adds Bones Howe, whom Waits had discarded as his producer. "She really separated him from everybody. I don't know if it was her personal jealousy or what. I don't harbor any bad feelings towards her, because I really believe she saved his life."

For the few who saw Brennan as a Yoko Ono implanting herself in Waits' career, many more saw the galvanizing effect she had on his work. "You can't really overestimate how much she brought positively to the table in a creative sense," says Michael Hacker, who'd worked with Brennan at Coppola's Zoetrope studios. "As great as his early stuff is, his new music was so revolutionary."

The new Waits music, which began taking shape in 1982, was fashioned out of diverse and disparate ingredients. Written substantially in a feverish two-week spurt on a second visit to Ireland, the tracks that came to make up Swordfishtrombones followed a rough narrative trajectory. "The songs have a relationship," Waits explained. "I tried to get them to knit." For his new work he used images rather than moods or characters as starting points for songs. The almost cinematic approach to writing seemed to crank open a place in his creative brain that he hadn't previously accessed. He was, he said, "trying to arrive at some type of cathartic epiphany in terms of my bifocals."

'Shore Leave', a song demo'd to illustrate Waits' new sound, featured the obscure metal aunglongs, which blended with marimbas to provide a murky undertow that offset the jagged interjections of Fred Tackett's guitar and banjo – not to mention the high-pitched sound of a chair being dragged across the floor.

Along with 'Frank's Wild Years', whipped up with the aid of organist Ronnie Barron and bass player Larry Taylor, and the mighty '16 Shells from a Thirty-Ought Six', 'Shore Leave' was what Waits played for Elektra-Asylum's Joe Smith one afternoon in April 1982, only to watch the blood drain from the veteran music man's face as he listened.

Smith, who in the late '60s had signed the Grateful Dead and Van Morrison to Warner Brothers, was now running a company whose flagship artists were the Eagles, Queen, and Carly Simon. Where the pre-merged Elektra and Asylum labels had focused on prestige acts from Judy Collins to Jackson Browne, Smith's brief now was to expand and build the brand within the Warner Music Group. Waits felt ever more out of place in this corporate environment.

"You have to understand that my eye wasn't on Tom," Smith says in his defense. "I was trying to keep Freddie Mercury from falling apart. I was trying to get Don Henley and Glenn Frey into a studio together. My promotion and marketing staff would roll their eyes when I announced we had a new Tom Waits record. I'm playing Queen and Jackson Browne, and we have a meeting and we play Waits and that's when everybody goes to the bathroom."

After some discussion, Smith decided to let Waits finish the album, hoping there might be friendlier music on it. "Tom and Kathleen decided they could make it better themselves," he says. "There was no point in battling that out because I didn't think another producer was going to up him to 300,000 sales."

"It wasn't like I was at a crossroads and asked myself, 'Am I going to go down AM Boulevard or Eccentric Avenue?'" Waits later reflected. "It wasn't that simple." With hindsight it probably was that simple: either Waits went out on a limb now or he stayed in his '70s rut forever.

Barney Hoskyns' Lowside of the Road: A Life of Tom Waits is out now, published by Faber & Faber