Facebook campaigns were launched to get them voted off before they'd even sung a note. Messageboards were bursting with complaints that they couldn't sing or dance, and had no place in the competition.
What a difference a few weeks makes. The 18-year-olds from County Kildare, Ireland are now mobbed by screaming fans whenever they venture outside the X Factor house in London and reportedly received the highest percentage of the public vote after their performance of Ricky Martin's She Bangs last week.
Even Simon Cowell, who branded them his "worst nightmare", has confessed to a "growing suspicion" they could win the competition.
The music mogul was bemused when his act Danyl Johnson was placed in the bottom two by the public vote at the weekend while the brothers sailed through to the next round with their performance.
"Simon is desperately worried that the twins are going to win," says their mentor Louis Walsh.
"He's really unhappy, but there is nothing he can do because this is who people are voting for. I thought they were going to be popular, but I never expected this."
Louis puts the twins' success down to their "likeability factor". He says: "They are sweet and naive boys. They asked Robbie Williams to their birthday party and wondered aloud if Britney was going to watch their performance."
And with their shaky dance routines, sky high quiffs, and cringe-worthy comments, critics have hailed them as "the best TV entertainment since John Sergeant murdered the tango on Strictly Come Dancing last year".
Whether they make it through to the final remains to be seen. But industry experts are saying it doesn't matter where they're placed in the competition – the siblings could easily make a million pounds from TV, panto, endorsements and appearance fees.