The 69-year-old hopeful faced fashion designer Julien Macdonald and his partner Janette Manrara in Sunday night's dance-off.
Tony, who performed the Charleston with Aliona, thanked his mentor and called the Kazakhstan-born dancer a "wonderful teacher".
"I'm a tough pupil," said Tony. "Aliona is a wonderful teacher. I'm going to stick to golf I think. I have enjoyed myself, it's been a glimpse into another world which I have thoroughly enjoyed.
"I had no concept of the work that goes into this, and the work ethic that these girls have. I did my best."
While Tony's fans at home voted for the golfer, and head judge Len Goodman called the sportsman one of his "golfing heroes" of all time, the scores were not enough to keep the British legend in the competition.
"Tony you've been one of my golfing heroes all of my life," said Len. "Sadly I have to say I would have gone with Julien and Janette, but you're still one of my heroes."
Fellow judges Darcey Bussell, Bruno Tonioli and Craig Revel Horwood all gave their verdicts, choosing to save the Welsh fashion designer.
Both couples performed their dance routine a second time on Sunday's episode, with Tony and Aliona doing the Charleston to It Don't Mean A Thing (If Ain't Got That Swing) and Julien and Janette performing the Tango to Applause by Lady Gaga.