"His wedding was his excuse for not having me in his life anymore," said the star, who eventually finished as a runner-up on the show to Joe McElderry, but has gone on to sell millions of records and has just released his fourth album.
CLICK ON PHOTOS FOR GALLERY
Olly Murs hasn't spoken to his twin brother since 2009
"He got into a relationship with his now-wife, and I don't think she ever liked us as a family," he told the Radio Times.
"For 25 years of my life, I always came first in his life and he always came first in mine. I loved my brother to bits. Anything he needed or wanted I'd do for him.
"But at that point in my life I was like, as much as I love him, I have to do this for myself, I have to change my life. I never expected him to react in the way he did."
Olly Murs' twin brother Ben at his wedding to wife Amy
Olly also revealed that Ben hasn't spoken to his parents either, adding that their mother would be "over the moon" if he did get in touch.
"His reaction was to block us all out of his life," the 30-year-old said. "He's living a new life with his new family. He doesn't want anything to do with us. He doesn't want to know us.
"I would love my mum to have her son back. She gave birth to both of us and she now only has one son. She'd be over the moon if my brother rang her and had a conversation with her and went for a coffee. It would change her life."