But the softer side of Simon Cowell was evident this week as he met a group of sick kids at a children's hospice.
The music mogul, who has been named patron of Children's Hospices UK - which represents a network of 44 children's hospice services - says he loved being around youngsters.
"They're funny, they've always got opinions and you can learn a hell of a lot more from them than you do from grown ups," he said.
"I just like being around them, maybe because I haven't got kids. Maybe because I'm still a kid myself, I don't know."
The 50-year-old has long been involved in working with children's charities.
And as revealed by HELLO! recently, he has developed a close bond with Sian Tolfree – a girl who suffers from a serious heart condition.
As Simon was busy putting a smile on the children's faces at Christopher's Children's Hospice in Guildford, Surrey, his fellow judge Dannii Minogue had her own special assignment.
The glam brunette switched on the Harrods Christmas lights with a little help from ten-year-old Dominic Head - a patient from London's Great Ormond Street Hospital.
This year's X Factor charity single – a recording of Michael Jackson's You Are not Alone - will raise funds for the hospital