The Made in Chelsea star spoke highly of members of the Essex-based rival reality TV show saying "we really like them, they are nice people."
Appearing on Loose Women on Friday morning, the 25-year-old was quick to address assumptions that his fame and fortune came easily.
"It slightly annoys me when people assume that I've never worked a day in my life," he said. "I've held many jobs, I've worked since I was 15.
"I've always had to work to pay my bills. My mother and father have always drilled in to me that work is very important.
"I'm very family orientated and I've worked very hard for money in the past."
The Eton-educated TV personality spoke on the show ahead of the release of his book Confessions of a Chelsea Boy.
The former PR manager hinted that the biography will be a no holds barred account of his life.
"I'm not a particularly private person, it doesn't bother me when things about me come out," he said.
"You will see a slightly different side to me," he said, referring to his on screen persona. "I don't believe I'm a villain, I think that every show needs certain characters in it."
"It [the book] only really touches on the show towards the very end. It's more about my growing up in a different way than many people have."
"Obviously I had some help with structure, I'm not an author," he added.
Referring to the BAFTA-winning TV show, for which he insists "there are no lines, there is no script", he explained how he handles awkward situations captured on screen.
"I'm not one to complain most of the time, the things that have happened to me have been as a result of my own actions," he admitted.
One of the most memorable scenes from the five series of MIC was when he was slapped by newlywed Millie Mackintosh in a London nightspot. "I deserved it," he confessed.
"I was not invited Millie's wedding… but I wish them all the happiness in the world."
Speaking about his own relationships, Spencer revealed that he struggles with commitment. "I'm 25-years-old and I find it very difficult to hold on to a relationship at the moment"