It's an airless night in Manchester's Albert Square. In front of the neo-gothic grandeur of the city's Town Hall sits a huge marquee. Inside the tent, a sweltering crowd gasp for air and gulp down Warsteiner beer. They are awaiting the much-touted/over-hyped (depending on your opinion) band Money, who are about to play the second of two sold-out shows as part of the Manchester International Festival.

Suddenly, a voice begins to sing from the back of the tent and instantly silences the boiling chatter. Money's frontman Jamie Lee emerges from the gloom. Reciting a song called 'Paradise Is Hell' – a gorgeous confessional delivered with a knowing smile – he meanders through the audience, randomly hugging and kissing bemused men and women, before settling down in front of a piano (again, alone and at the back of the room) to perform 'Goodnight London' - a eulogy to the lonely and disposessed.

When Lee finally joins his bandmates onstage (guitarist Charlie Cocksedge, bassist Scott Beaman and drummer Will Byron make up the quartet), Money pick a path through their compelling debut album, The Shadow Of Heaven. It's a set full of songs about love, loss and faith - sad hymns hell bent on reclaiming the word 'anthem' back from the gross distortion of stadium supergroups. Tracks like 'Bluebell Fields', the epic pop of 'Hold Me Forever' and the bleak luminosity of 'Letter To Yesterday' fizz with visceral intent. Lee is the focal point; cocksure, ever-so-slightly unhinged and a reet charmer. There is a sense watching him, and watching Money, that a very special night has unfolded. At the end of their set, the band hand out small scraps of paper inscribed with the words 'Manchester is Paradise'. For just one night, they may well be right.

Two days later, I meet up with Lee in a Manchester city centre bar. He tells me he's not feeling "especially lucid" and I suspect he hasn't slept much since the Albert Square shows, which were the biggest UK gigs the band had played. Even in a sub-lucid state, it's apparent that Lee possesses both intelligence and a healthy streak of curiosity. He asks me a barrage of questions about my background, family and career, and seems genuinely interested in my responses.

Why did he choose to start Saturday's gig by singing alone amongst the audience? His answer isn't what I anticipated, and sets a pattern for our conversation. "To try and be slightly romantic about it, and I don't know if I can get away with saying this or not," he says, "in a way I've given up on certain things. I've stopped caring so much. I have decided to do whatever it is that I want to do. So, if I write something I like – like 'Paradise Is Hell' - I want it to be heard. If I am being less romantic, singing acapella at the back of the room makes everyone immediately quiet."

Boom. Within thirty seconds, we are straight into some potentially heavy shit. "Why have I stopped caring? Because I lost faith in all those things you have faith in when you are young." Like what? "Faith in all things," he replies. "Some stuff happened in my life, as it does in everyone else's, which was very disappointing. You can either try to regain your faith or you try and desecrate every speck of it. It's an experiment – I'm trying to live faithlessly. I want to make a perfect mess."

I begin to realise that this unashamed self-analysis is typical of Lee and is, in part, what makes Money either one of the most exciting new bands in the country or – to the unconvinced – a set of half-baked chancers. Having fallen in love with The Shadow Of Heaven, I'm a believer, but part of the thrill about Money is that they will engender divisive opinions. The bombastic beauty of their music, the outpouring of prose – Lee's opaque monologues for the unloved – and a desire to shock, will undoubtedly enrage as much as it will captivate. Their website address is lonelysexydeath.com, for chrissakes.

But, of course, Lee won't care about public opinion. "I'm willing to be wrong and fallible and weak," he tells me, before asking, "do you mind if I go for a wee?"

Money were formed in Manchester's studentland, via a mutual love of books, films and music. Their debut single was released on Sways Records, the self-annointed 'cultural regenerator' from the neighbouring city of Salford. I ask Lee about the impact Manchester has had on Money. As a forty-something Mancunian, I settle back and await a generic cap-doffing tribute to bands of yore. "Manchester has given us everything," he begins. "I am feeding off it. I like to see Manchester in mythological proportions. In a way, the people that live here are symbolically dead, as in they don't seem to have the same cares as those living in a fearful way, and they are able to do whatever they want. Having lost their mortality, they are celebrating.

"Also, perhaps because of the geography, you are forced to acknowledge and see people from very close up," he continues. "Manchester is very honest; it's very hedonistic, artistic and poetic. All of these things are out in force here and you can touch them – they are right in front of you."

At this point, Lee glances at my notebook. He can see that we've covered the word 'Manchester?' and eyes up the next subject matter. "You've written 'Sex?' – can we do the sex question?"

Money seem very comfortable with their sexuality; the merchandise stall outside the gig was doing a roaring trade in t-shirts featuring a full-frontal picture of Lee, and the man himself seems refreshingly open about his bisexuality. "I was sexually active at a young age in a slightly devious manner," he says. "Liking both men and women has made me question, and be inquisitive about, other aspects of my life which are not sexual at all. That small theatre of uncertainty inside myself has created uncertainty in every aspect. For a young person, I think it is quite difficult to have the knowledge that there is some otherness about you, which is very personal, but you realise you cannot express. But, my bisexuality gave the curiousity I already had some colour and created certain aspects of my personality."

Although the other members of Money are talented musicians, who all contribute to the band's democratic songwriting process, there is no doubt that Lee's charismatic personality is central to the band's allure. On stage he fluctuates between being flirtatious, comical and pin-droppingly intense. Is that persona an act? "I'm being drunk, cocky Jamie," he replies. "I enjoy being onstage, but I don't want to get to the point where it becomes buffoonery, and I'm conscious I'm treading close to that line on occasion. I don't want it to detract from the music."

But Money are not just about the music. Lee also writes prose and poems, and from an outside perspective these words appear to form an important part of the band's worldview. It seems, however, that I am wrong. "I hope the writing is not connected to the music," he emphasises.

Why not? "The difference between a great piece of writing and a great song is that writing can have near-perfect clarity, [while] a song is a mystical phenomenon which doesn't have to be particularly precise. I know it sounds terribly self-indulgent, but I'd like – at some point – to be considered as a writer and less as a musician. But I cannot really think of anyone who was branded as a musician and then was able to become a writer. It seems you have to be one or the other. Musicians don't seem to have the intellectual authority that writers have."

So where will Money's artistic direction take them in future? "All the lyrics on this record were written to music," Lee reveals. "I would play guitar and then fit the words around the melody. For the next one, I am trying to write poetry – using stuff I had written when I was having a really hard time – and take those ideas and impose them on the music, rather than the other way around. The next album is not going to be a populist record at all."

As our chat draws to an end, I'm still intrigued about the implications of Lee's initial statement that he has "stopped caring." Isn't he treading a dangerous path? "Well, there are some unhealthy implications, such as my relationships and my own health. With regards to the band, everyone has their own ambition and that needs to be respected. I will write down a quote that I think sums my attitude up."

The tape recorder is then switched off. We continue talking for a while and are joined by a couple of Lee's mates and a young woman, who, even though she is on an actual date with a burly chap, has decided to make a play for Lee. He asks her a barrage of questions and within minutes she has gleefully recounted her life story. Lee's curiosity, it seems, is charismatic.

The next day I flip through my notebook. Under my questions lies that quote from Lee. I hadn't noticed him writing it down. "Beauty is the thing that knowingly embodies the very process of its own destruction, and having acknowledged its inevitable demise, then begins to do so with an element of grace and begins to dance."

Money's album The Shadow Of Heaven is out now via Bella Union