“One of the nicest and most devastating things ever said to me,” says Clive James of his on-off songwriting partnership with Pete Atkin, “was, ‘What you’re trying to do is too good.’ ” Happy both to inflate his own balloon and prick it, he adds, with that smirk familiar from his television persona: “I walked away foolishly congratulating myself, then thought, ‘Wait a sec, he means we’re dead as doornails.’ ”

If only they were French; a musical career would have been so much easier. Starting out as Cambridge Footlights in the late 1960s, they made six albums - Atkin setting and singing James’s wry, lovelorn, occasionally clever-clogs lyrics - before punk kicked them into touch. Despite a cult following, which is now thriving online, UK record companies never knew what to do with them: were they folk, revue or easy listening? In France, they could have poured each other a pastis and smiled: “Vive la différence.”

“People who write and perform our kind of stuff can make a living in the boîtes and on the radio,” James says. “Brel and Brassens and all those boys: it’s a tradition over there, mainly because popular music is quite weak. Carla Bruni is another.” He winks: “She’s somewhat obscure now, because she married the president and dropped out of the headlines, but that first album of hers is wonderful.”

Although James allies himself and his musical partner with chanson, their material is hardly foreign to British or American ears. A clue to where it comes from is there in the subtitle to Atkin’s current album, Midnight Voices: The Clive James - Pete Atkin Songbook Volume I, recorded largely because the pair don’t own the rights to their original LPs. The key word is “songbook”. James, now 68, and six years Atkin’s senior, has never been a real rock’n’roll animal.

“This wheeze was new to me/And caught my sense of fitness on the hop,” he writes, in 1973, in his verse letter to Atkin, a kind of retrospective manifesto that can be found on Atkin’s website. While admiring “postPresley pop” for its “simple verve”, he slates it for being “well pleased with the banalities it blurted out”. As a lyricist, James’s true heroes and forebears remain the likes of Oscar Brown Jr, Johnny Mercer and Cole Porter.

Posterity alone will decide which, if any, of such elegant, nuanced numbers as Beware of the Beautiful Stranger, Between Us There Is Nothing or Be Careful When They Offer You the Moon enter the repertoire of “standards” that has furnished crooners from Bing to Bublé. In part, however, the intention is that they should. According to James: “The ideal is to write a song that someone else might do in a different way.” Slightly embarrassingly, though, the biggest royalty cheque Atkin and James have received so far came when Val Doonican, the light-entertainment king and knit-wear model, covered The Flowers and the Wine in 1981.

Every genre has its standards - country, folk and soul, especially. They are, or should be, the most enduring achievements of the form, killer tunes coupled with words that strike a chord, time after time. Disposable pop is a cheap thrill; standards, by their nature, age well. As Jon Nakagawa, who programmes the American Songbook concert series at the Lincoln Center, New York, says: “A standard needs to be covered by other singers to be defined as one. If the song dies with the original singer, it can’t be a standard.”

Plenty of pop tracks, notably the Beatles’ canon, have become standards in their turn, but the term also implies a historical moment - roughly from the “big bands” to the “bobbysockers” - and certain characteristics. A check list of those traits would read: urbane wit; lyrics and melodies that go together like Sammy Cahn’s horse and carriage; solid rhymes; stylishness as crisp as a freshly ironed shirt; and a decorum in delivery that allows you to hear all the words.

Nakagawa’s bills give currency to the classic American songbook and make new entries to it, too. As well as the opera star Deborah Voight singing show tunes, and Patti Smith the songs her mother sang, Sufjan Stevens and Neko Case have performed under his banner, as has the Magnetic Fields’ Stephin Merritt, a man who named his dog after Irving Berlin. Merritt’s absurdly accomplished 69 Love Songs, melding Tin Pan Alley, indie rock and electropop, is an album every home should have. “Most of what we believe follows the tradition of American popular music is lyric-driven and tells a story,” Nakagawa says. “But only time will tell what sticks and what doesn’t.”

What gave James the hump in the early 1970s was that the qualities of the “complete song”, which is what he and Atkin were striving to produce, were overlooked. “Only a few of the critics, exemplified by the wonderful Dave Gelly, wrote about my lyrics and talked about my craft,” he says. “Most rock critics either didn’t notice it or thought it was inimical to what popular music should do.”

More than any matters of taste, it was the decline of craft in pop - in James’s view - that was the problem. It wasn’t that one genre was superior to another, but that songwriters in general got sloppy. “Leaving it loose was something that happened only after rock’n’roll,” James says. “There were people running round, back in the day, who weren’t really writing lyrics, they were fitting random words together to quite pretty melodic and harmonic structures.”

Then, as now, there was a glut of singer-songwriters putting pen to paper; and that first wave, James believes, swept away many of the virtues of the well-crafted song, changing the pecking order in the music business. “When singer-songwriters could earn their money twice [from performing and from publishing rights], the old songwriting partnerships were no longer on easy street,” he says.

Posted on Atkin’s website are copies of articles James wrote for Creem [sic] magazine that, nearly four decades on, still get the Geiger counter of controversy buzzing. They might seem a bit cranky - “The attitude of those essays is not one I would repeat now, but I thought I was preaching in the jungle,” he chuckles - yet they remain, for rock hacks as well as fans, terrifically bracing, especially the one about Bob Dylan.

“There is never a Dylan song where the whole song is as good as the best stanza, or the best stanza is as good as the best line,” James maintains today. “But you’re operating at a stratospheric level. You only have to read his prose memoir to see the guy is a writer all the way through his body. Obviously, they are writers’ songs, but I think my criticism of them is valid.”

Even as a vehement Dylanite, I find it hard to disagree. And it occurred to me, watching Adele cover Make You Feel My Love on the inaugural Later Live... with Jools Holland, that Dylan has regained his mojo lately by writing latterday equivalents of bluesy standards. Is anyone else doing likewise? Atkin and James both rave about Joe Henry, who was part of this year’s American Songbook season. What about him?

“Joe is a songwriter who upholds the ‘form’ of songwriting that leads to ‘standards’,” Nakagawa says. “He writes beautifully crafted songs, but I don’t think people today set out to write ‘standards’ like they did in Tin Pan Alley.”

In the era of The X Factor and the coffee-table jazz chanteuse, though, songs that can be sung convincingly in evening dress aren’t going away. While I can’t imagine Leona Lewis, whose Bleeding Love was recently described as a future standard (please, please, no), getting her pipes around a line of James’s such as “Touch has a memory/ Better than the other senses”, maybe Diana Krall could.