"The truth is -- and I am not being rude -- I wouldn't hire Gary to run my record label," Simon said in a recent interview.
The new dad also told The Sun: "Gary was seen as the head judge - and so am I. I do believe I am the best person at spotting a potential star and making them work well on the show.
CLICK ON PHOTOS FOR FULL GALLERY
"That is what I am good at doing, not being a big-mouth on the show. It is making the show I am on better."
Simon was replaced by the Greatest Day singer as head judge when he left the X Factor in 2010 to work on the US version of the show, and he blames the 43-year-old for a huge slump in the TV talent show's ratings.
The three series seeing Gary as head judge ratings have faltered, with last year's final, won by former prison officer Sam Bailey, reaching its lowest viewing figure since 2005 with an audience of 9.7 million.
Simon will return to the judging panel for the new series of the show alongside former judge Cheryl Cole. Irish funnyman Louis Walsh is also expected to return, while a fourth judge is yet to be confirmed.
Simon said: "I think all these shows have become so celebrity driven, you are hiring someone based on how famous they are and I am not sure that is a good thing in the long term.
"Essentially that is what you are doing with The X Factor judging panel. But if Gary and I were running a record label we would do it in a different way."