Simon is also back on the show for the first time in years, after attempting to take the show to the United States, with Britney Spears and Demi Lovato as his fellow judges.
On Monday, the four X Factor judges arrived in Manchester for the very first auditions of 2014, and Louis, who has been with the show for the entire 11 year run, reportedly took the chance to wind up the female judges.
One of the first acts on stage were a girl group, and Louis decided to ask them which band they preferred - Girls Aloud or the Spice Girls.
"They said they liked them both so any huge row on the first day was avoided. Louis was giggling because he knows everything won’t stay harmonious the whole way through and they will have differences." a source told The Mirror.
The source added: "Part of the fun is watching the relationships change over time and seeing different opinions between the judges."
Part of the fun of the show, especially during the audition stages, is watching the novelty acts and the source revealed that one auditionee's mum decided to join her - with a large Chinese banquet.
Mel B and Cheryl Cole at the first X Factor 2014 auditions
"She had brought a huge Chinese banquet with her and a cake for the judges. Simon saw the funny side and tucked in with gusto. But it meant there was a 20-minute hold up for the next act waiting to sing. It didn’t do their nerves any good", said the source.