The music mogul's mother Julie Cowell has revealed she watches every episode of his TV talent show The X Factor. And she's straight on the phone afterwards to give her verdict.
"I tell him if I think he's said the wrong thing to a contestant," says the 82-year-old.
Julie also added that Simon's first put-down came at the age of five, when he criticised her new hat.
"I asked him if I looked nice in it," she explains. "He replied: 'You look like a poodle'."
However, she thinks he doesn't deserve his 'Mr Nasty' nickname. "Simon isn't nasty – he's just honest," she insists.
"I've always insisted on politeness, so when people say how pleasant Simon is, I mutter 'thank goodness'. It means more to me than his television career."
In Yours magazine, Julie goes on to say how Simon, who split from his long-term partner Terri Seymour last year, has always had firm views on settling down.
"He said at 19 that he wouldn't get married. I though he might change his mind, but he's now 50 and it doesn't look likely to happen," she says.
And what do you get the man who has everything for his 50th birthday? "When I asked him what he wanted, he said, 'nothing'," says Julie, who in the end got him "book tokens and his favourite after-shave."
She adds: "Not very showbiz – but away from television Simon is just Simon, the same person he's always been."