Referring to his split from his wife of 15 years in the summer he said, "But the people that are judging me are judging me without knowing the whole story."
"Nobody knows the whole story. There's two people that know the whole story. The people that matter know everything.
"The people that don't matter make up their own things and then have an opinion. Their opinion they're entitled to. But you have no right to put it in a paper."
He spoke about the rumours of him having an affair with his 34-year-old American Baking Competition co-star Marcela Valladolid circulating in the press.
"Nothing really stuck. Nothing. Because at the end of the day it was all conjecture, hearsay, alleged gossip. There was nothing ever there."
When questioned about the nature of their relationship he told the Times, "I'm not even going to talk about it. I never comment on private life."
Responding to the journalists claim that even the press want to hear that he and his wife Alexandra overcome their marriage problems he replied, "But how can that happen when it's in the press? It can't happen because the whole thing is out there.
"You have no idea what's been said, how many times I've picked up the phone, spoken to Alex. We've had chats, but at the end of the day, it's my life, it's her life, and it's our choice."
The Liverpool-born TV chef also opened up about how he and Alex are getting on now.
"I was with her last night. We had a nice meal. I only live a mile away. I've got a nice barn. It's lovely. Josh stays all the time. She comes over a lot.
"I'm a single man and I'm happy doing what I'm doing at the moment. At the moment, even if I was still with Alex and nothing had happened, I couldn't see her anyway because I'm away all the time.
"What happened has been unfortunate, but you don't know what's going on. Nobody does. But I'll never say 'never'. I'm happy being single at the moment. I'm busy. So I'll just concentrate on my work and make sure Josh is all right."
The 'George Clooney of baking' left his marital home in May amid rumours he had started dating Mexican pastry chef Marcela Valladolid.
Divorced mum-of-one Marcela, the US version of Mary Berry, and Paul are said to have "hit it off from day one" with Tequila brand owner Marcella describing him as a "handsome devil".
After coming back to the UK, Alexandra said the dynamics of their marriage had changed.
"I feel utterly bewildered and incredibly sad. Paul went to America and when he came back everything was different, everything had changed," revealed Alexandra.
"I'm distraught, completely devastated, but I don’t want or need revenge," added Alexandra. "I just want us both to be able to rebuild our lives for the sake of our son Joshua, because he is the most important person in our lives."