Robert Downey Jr. may be off the cocaine and on to nicotine and calcium, but his conversation still takes some surreal turns

There is, very likely, a deliberate irony in the title that Robert Downey Jr. has chosen for his first album. It's called The Futurist, not, I'd guess, because he is interested in a half-forgotten art movement, but because he knows everyone thinks only of his past. It is not so long ago that Vanity Fair compiled a profile of one of America's best but most troubled actors.

Our venue, mercifully, is his record company's hip little studios in Santa Monica. Here I find Downey - still Hollywood handsome but, at 39, crinkling round the eyes and whitening at the temples - jabbing a fork into a salad. When he has finished, its foam container is converted into an ashtray. Addictive personalities never become non-addictive, but the successful ones divert their obsessions to lesser evils. His, I would guess from his wiriness, are nicotine and exercise.

He has just come from the gym, where the TV has been showing a mocking countdown of celebrity misdemeanours. When he left the programme had reached No 70. He assumes that his own would have ranked higher. And which would they be, I ask. "Maybe the Palm Springs shenanigans," he says, referring to his arrest during Thanksgiving 2000. "Or maybe the Goldilocks incident in Malibu." In 1996 neighbours found him, after a particularly confused night, asleep in their 11-year-old son's bed. "They have a cornucopia of choices because public humiliation until recently was a big theme of mine."

Did he feel humiliated? "I felt like Sir Walter Raleigh a bit, except that I didn't bring tobacco to the New World or do anything particularly innovative. I was just kind of dizzy, wondering what the hell was going on."

Yet not, he concedes, all the time. In film after film, from Short Cuts to Wonder Boys, he pitched up on set sober and on time ready to deliver high-definition performances that tended to leave his co-stars looking feeble. In 1992 he was nominated for an Oscar for Chaplin and, with better luck, he might have been nominated again.

"Through all that tough stuff I stayed rooted enough to be able to complete projects, father a kid and purchase furniture," he says. And he was married for several years. "But I wouldn't recommend that to anyone, unless it was exactly what they needed, but, you know, long periods of time in something fairly akin to purgatory is a great remedy for a life that's short on gratitude."

At such moments in our hour together I wonder what Downey is on about. At others, as he gabbles away, pummelling cigarette stubs into the salad shreds, I wonder what he is on. He insists nothing, but that's not quite true for, suddenly, he opens a canister of multicoloured pills and swallows a handful. My God, what are these? "Herbs, calcium, E vitamins."

How does he get his highs, now that he's a narcotics-free zone? Mainly, he says, from recalling how close he came to writing himself off. "It's like the thrill you get from escaping a car crash. For those 15 seconds afterwards you're so appreciative. Adrenalin is shooting up your meridians and you go 'Wow!' Ten minutes later you are beeping the prick in front of you."

Coincidentally, he says it was while driving a real car a couple of years ago that an epiphany occurred. He was not on probation or parole, not in a relationship. He was free, in other words, to misbehave. So there he is, driving his Lincoln in Venice, California, when a cop stops him. On his front seat, poking out from a bag, is his paraphernalia, "nitrous oxide, glass stems, pipe screens, the whole thing." And the policeman doesn't look in the bag, just tells him to fix his front number plate. A couple of days later he is driving along the Pacific Coast Highway, still not having used the kit, and he cottons on that the universe has given him another chance.

So he threw the drug stuff out? "Well, it found a happy home." Nothing goes to waste? "More the material itself went to waste," he says, suddenly censorious of his former crowd. "It was imperative that I not interact with the type of people who'd want to pick up the slack. So now I can say with some assurance, not with the bravado or bullshit that comes with the untrustworthy alcoholic archetype, that - unless I feel the need to start writing a sequel to the whole story, which I don't - it's behind me."

The whole story is a lengthy one and began soon after he was born into a family of 'beatniks' in Greenwich Village, New York. The family moved often, heaping more confusion on to their son's life. In California he dropped out of high school. He found in his chosen career as an actor that he could snort cocaine and drink all day without obvious malfunction. Aged 22, he played a coke addict in Less Than Zero.

Things began to unravel only after Chaplin. It is a tragic coda to a magnificent performance, for, should anyone doubt Downey's talent, they need only fast-forward to the moment when, as Chaplin, he arrives in the Californian desert and, impromptu, does his entire, splay-legged, umbrella twirling act for the director. In that moment Downey announced not only the tramp's genius but his own. Sadly, the applause the performance earned was not enough. As he saw it, it just condemned him to more hard work: "There's the high and there's the buzz and there's the achievement and there's the accolades and then you are really, really, really supposed to go right back into f***ing training again. That's the point that I missed."

The year after the Academy nomination Downey, who had spent seven volatile years with the actress Sarah Jessica Parker, married the model Deborah Falconer. The same year they had a son, Indio. Again, Downey failed to get the point, in this case that marriage was not the end of the story but a commitment to continuing it. As he puts it, he had difficulty putting his tux back in the closet. In 1996 he and Deborah separated. In 1997 and 1998 he spent a total of 113 days in Los Angeles jails. In August 1999, after more parole violations, he was sentenced to three years and spent nearly a year inside. When he was released he was, frankly, lucky to be cast as Ally McBeal's fiancé in the TV series. Of course, he charmed in it, screwed up, and was fired.

"You know what? I wasn't unhappy doing the show but there was a director who pissed me off. I'd been out of prison for three months and I recall going into the dressing room, dialling 9 for an outside line and I remembered this drug dealer's number. And when I finished work that day I quietly said 'Thank you' to this director whose guts I hated and got in the car provided by the salary I was making on the show and went out and that was it. Those were the kind of lizard-brain decisions I was making."

The press, unusually, had largely been rooting for Downey, but now it asked why nothing was working. Did the Californian penal system treat celebrities with kid gloves? Should Downey have been allowed a day release to film an Elton John music video? Or, perhaps, jail was itself the wrong approach. If addiction is an illness, shouldn't it be treated, not punished? Downey thinks both points of view may be right, depending on the individual. "Most dope fiends love nothing more than hearing society say 'He's not hurting anyone else but himself. He's sick and shouldn't be incarcerated.' Because that's another out."

A third possibility is that in Downey's case the drug abuse was not so much an illness as a symptom. Psychiatrists have before now diagnosed bipolar depression, which he doesn't dismiss out of hand. The term might, for all I know, fit both the nervy, garrulous man I meet and the dissatisfied, ennui-prone one he describes when he talks about his restlessness.

"I'm like the kid who doesn't want to go to school so pretends to be sick. Maybe sometimes he is really sick, too sick to go to school. And what's so hard about a day at school? But he needs something extra. Have you ever had that feeling that you needed something more than the people around you just to be OK? An extra five minutes in the shower? A better breakfast than just whatever crap was in the pantry? I needed that something."

After the humiliation of Ally McBeal, his friend Mel Gibson hired him to play the title role in the remake of Dennis Potter's The Singing Detective. Even held against Michael Gambon's performance as the head-to-toe psoriasis sufferer, Downey was brilliant, turning from sincerity to sarcasm on a sixpence, lacerating himself and everyone else. He thought a lot about the screenplay's meaning and, perhaps significantly, finally brought it down to a medical level. As Potter had said, it was a story about a sick man who picks up his bed and walks.

The Futurist has played a part in Downey's own decision to pick up his bed and walk. Half-rock, half jazz, the songs, with their obscure yet beguiling lyrics, are, with two exceptions, written by him. They are not heart-on-sleeve confessionals but personal all the same. Was he thinking of Sarah Jessica Parker when he wrote 'Details', in which he hypocritically promises to spare his lover the details of "the rocks and nails, the times I have lied"? "Definitely. A big part of the song was an apology for the failure of that relationship at my hands." 'Broken' and the title track, conversely, are about commitment. They would not have been possible, he says, had he not met Susan Levin, one of the producers of his weird 2003 movie about female lunatics, Gothika. They are to marry in August.

"Susan's really grounded, supportive, a hard worker, a great judge of character and really motivated." So his opposite? "Sure, but she has definitely put a new spin on what I thought was possible. I'm not saying that I put up with duds, I had great women, but somewhere along the line it was pretty much revealed to me that you can't trust them. Along the line that became, I thought, one of the secrets of the universe."

I say this sounds very 'Singing Detective'. Its protagonist, Dan Dark, is convinced his blameless wife is cheating on him. He agrees. "And at the time people were saying 'You do see the correlation between you and Dan, don’t you?' I was like, 'Well, he's a great character but are you kidding me?' It never ceases to entertain me how naivety and denial just make for dinner and a show for me."

And so, after all, it remains a matter of tense. Has denial truly receded into his back mirror, or could it still wreck the futurist’s future farther down the road? I leave him vamping away on the piano, his feet at the pedals, pumping neurotically.

The Futurist is released by Sony next month.