A strange tale: after signing with 4AD in 2002, The Mountain Goats – aka John Darnielle, a prolific purveyor of magical realist narratives, coated in otherworldly tape hiss – began to literally clean up his act, recording with a full band and professional studio set-up. The result was his most satisfying work yet: the claustrophobic Tallahassee, the damaged, heart-breaking We Shall All Be Healed (a concept album about his years as a teenage meth addict), and The Sunset Tree, an album both darker and more defiant in its exploration of his abusive childhood.

You might think that a man so well-exorcised would be running on empty, but Heretic Pride comes up with riches, by mining a seam of vividlyrealised fiction. A collection of short stories or character studies – think of the snapshot constellations of Raymond Carver – with perhaps the only thing binding them together being the characters’ status as freaks, misfits, in some cases outcasts from the others’ stories (HP Lovecraft, Sax Rohmer). ‘San Bernadino’ sees an unmarried couple escaping to a motel with their new son; ‘In The Craters On The Moon’ is populated with recluses haunted by disasters, awaiting their extermination; the title track describes an execution-by-angry-mob, the narrator being beaten and then set alight to the soundtrack of an upbeat, rolling, piano-and-organ-flecked groove.

It’s Darnielle's way with ambiguity, best evidenced here, that gives much of these songs their magnetic power: the shattering ‘Marduk TShirt Men's Room Incident’ sets the discovery of a corpse, the narrator comparing her with his exlover, against skeletal acoustic, delicate strings and the cooings of the Bright Mountain Choir; it's hard to say whether the narrator of ‘Michael Myers Resplendent’ is an actor preparing for the role, or the serial killer himself anticipating finally playing his own part. It’s disappointing that the accompanying press release – drawn by Jeffrey Lewis – so bluntly pins down the songs’ meanings, because their suggestions, questions and lacunae – tales dead-ending, voices guttering out – are so much more powerful. The little details of sound and story – how appropriate Darnielle's weak, nasal voice sounds in these emotionally strained songs; the melancholic religious speculations of ‘Sept 15 1983’, about the death of Prince Far I – provide the reasons to love these songs.

If what you want is a lump in your throat, a smile on your face, and ideas in your head, there’s no one better around than The Mountain Goats, and Heretic Pride is possibly their best transmission yet. A strange tale, but true.

Daniel Barrow talks to John Darnielle

Every Mountain Goats album seems more lush and complex than the last. Are you hoping to attract more fans, or is there some other reason?
"I just do whatever seems most interesting and fun to me at the time – I think any other way of doing things would probably be a disaster. I'm sure there are people who're able to say, 'OK, how can I attract more listeners?' and so on but I follow a pretty instinctive process: write the songs at home, send them to the people I want to play them with, then see what happens when we get into the studio. These songs seemed kind of lively for the most part and we were really enjoying playing as a band so they came out like that."

A lot of these new songs involve other people's fictional creations – Sax Rohmer's spies, Lovecraft’s malevolent entities, Michael Myers. How did they end up in there?
"I partly blame this concrete room I rented to keep as an office since my guitars were sort of coloniing the house – I started going down to the office in the morning and sitting on the floor in there with my guitar, and I felt like I used to feel when I'd spend a few weeks of summer vacation visiting my father in Oregon: he had a full basement at his house and I’d hang out down there, sometimes chopping wood with an axe or reading science fiction paperbacks and wondering whether monsters were real and stuff like that. I think of this record as sort of an indexing of life-long obsessions."

The last few Mountain Goats albums have been largely autobiographical, but they’ve been written in a similar first-person narrative style, and exhibited many of the same feelings – lovelessness, pain, desolation – as this new one. What, then, is the border between autobiographical and fictional writing?
"Well, everybody seems to think Get Lonely was autobiographical but it really wasn't – it just sounded like it must have been, but those were just stories. The two before that, yes absolutely, in differing degrees. "Anyway, I think whatever borders there are tend to err more in favour of fiction – nobody's feelings occur to them in rhyming couplets or notes of the scale or even words, right? The line on writers is that they’re only ever telling their life stories in some way or another but I wonder if it’s not the other way around – that even people who’re trying to tell their stories are in the end only making things up to try and make sense of a lot of disorder."

Did you enjoy making this album as much as the last few?
"More! Adding John Wurster on drums was just awesome, and we'd toured with him earlier in the year so it wasn’t like adding an unknown quantity – and we had JV back in the studio so it was like a family reunion with this awesome new relative that only some of us had met. "Plus, these songs were just more fun to play than the last couple of albums – Get Lonely was like digging a tunnel and The Sunset Tree was this massive catharsis for me, but this one was like getting to hang out in a haunted house or something. Really fun."