Scroll down to vote for who you think will replace Gary.
CLICK ON PHOTO FOR GALLERY OF FAVOURITES
Simon Cowell is a strong contender and could be making a comeback. As a judge on the show for seven series, Simon worked on The X Factor in its most successful days before leaving in 2010 to launch The X Factor USA.
With the stateside ratings at its lowest ever this series, Simon may be considering a return to the UK show, and could be sealing a potential £50million annual deal, according to The Sun.
An insider added that Simon was eager to see Cheryl Cole back as a judge. The Fight For This Love singer was axed from the first season of the US show, but could be making a comeback on the original version if she decides to live in the UK long term.
Simon Cowell and Cheryl Cole could be making an X Factor UK comeback
Robbie Williams has also been tipped as a favourite. The Candy singer has already made an appearance on this year's series, assisting Sharon Osbourne at the judges' houses stage, and has also previously acted as a mentor to Gary.
The heavily-tattooed star has taken the X Factor stage by storm, duetting with previous finalists One Direction and Olly Murs as well as performing his hits solo.
Olly is said to have impressed show producers this year as Gary's guest mentor, and could seriously be making a return. The former X Factor runner-up enjoyed the show so much that he signed on as a host on The Xtra Factor, but left at the end of the last series.
Robbie Williams and Olly Murs have been tipped as front-runners
"We'd love to get Olly back as a young judge to freshen things up," an insider told The Mirror. "He can relate to the other singers like no one else."
Shane Filan also acted as a guest mentor to Louis Walsh this year, and could be following in his Westlife manager's footsteps and take on the coveted role.