The actor's name appeared on Lindsay's now-infamous list of lovers, and he has insisted on numerous occasions that he should not have been included.
This week, he admitted that he shared a "lame" kiss with the star around the time he was filming Spider-Man 2 in 2003, but was adamant it went no further.
"I will swear on anything that you ask that I have never had sex with Lindsay Lohan," he told Howard Stern on The Howard Stern Show.
"Here's the problem," he continued. "I bet you if we brought her in here and you asked her to her face, 'Did you have sex with James?', she'd say 'Yes'. I think she's that…"
"Delusional?" the host interjected. "Or doesn't remember," James replied. "Alright, we maybe kissed. It was lame.
"I can't believe she put me on that private list," he later added. "She's so delusional!"
It's not the first time the 35-year-old has spoken to Howard about Lindsay. Last year, he told the radio host he had previously turned down the actress' advances.
"I mean, I don't want to brag about it," James said. "I don't know how that got out."
"[Lindsay] was having issues even then, so you feel weird," he added. "Honestly, she was a friend. I've met a lot of people that are troubled and sometimes you don't want to do that."