"Nobody knows the whole story. There's two people that know the whole story. The people that matter know everything," he told the Times.
"The people that don't matter make up their own things and then have an opinion. Their opinion they're entitled to. But you have no right to put it in a paper."
In the same interview he addressed the affair rumours saying, "Nothing really stuck. Nothing. Because at the end of the day it was all conjecture, hearsay, alleged gossip. There was nothing ever there."
When questioned about the nature of their relationship he said, "I'm not even going to talk about it. I never comment on private life."
Divorced mum-of-one Marcela, the US version of Mary Berry, and Paul are said to have "hit it off from day one" with Tequila brand owner Marcella describing him as a "handsome devil".
After he came back to the UK, Alexandra said the dynamics of their marriage had changed.
"I feel utterly bewildered and incredibly sad. Paul went to America and when he came back everything was different, everything had changed," revealed Alexandra.
"I'm distraught, completely devastated, but I don’t want or need revenge," she added. "I just want us both to be able to rebuild our lives for the sake of our son Joshua, because he is the most important person in our lives."