The music mogul, who is expecting his first child with socialite Lauren Silverman, made the joke at a press event and indirectly alluded to the surprising news of his girlfriend's pregnancy.
"It's been ten years of The X Factor and this is the first time I've seen the edited show," said Simon. "It was absolutely incredible. We have a great panel, great talent, and it's just so easy after ten years to get lazy and to make the same show again.
"But the enthusiasm, the changes and the panel are fantastic. Dermot O'Leary of course, Caroline Flack, the new arrival — Matt, Matt, I can't remember everyone's names," he joked, referring to The Xtra Factor's new co-host Matt Richardson who has replaced Olly Murs.
"The show just feels fresh, fantastic, and I'm very proud to be part of it. Once again, I feel like a very proud daddy.
"I want to thank all you guys in the room who've slagged us off and written nice things, but it's all been great because it's made us successful and loads of money so thank you very much indeed," said Simon.
The first episode of The X Factor will air on Saturday 31 August, where viewers will get a taste of the preliminary auditions.
This year contestants will have to perform in front of the judges in a small room, before being selected to go through to the arena auditions where they will sing before a 4,000-strong audience.