The rap star is said to have declined the opportunity to be a part of the talent show for fear of becoming "too mainstream".
"He didn't want to lose any street cred," a source told Us Magazine. "So he turned it down."
Kanye, who recently became a dad for the first time, could have starred alongside Keith Urban and Jennifer Lopez, who is due to return to the programme.
However, there is one TV show that Kanye is happy to appear on.
On Kris Jenner's talk show, the 36-year-old will open up about fatherhood, his turbulent relationship with the press and his relationship with Kim Kardashian.
An insider told the magazine that the notoriously private musician was "really bringing it" for the interview as he chatted about Kim, their two-month-old daughter, North, and living with the show's host Kris.
The Gold Digger singer even makes a little joke at Kim's expense, saying that he's been "so in love" with Kim, who he started dating in April 2012, that he thought about "playing sports " — Kim was previously married to basketball star Kris Humphries and dated NFL player Reggie Bush.
Kanye's hour-long interview will mark the end of Kris' trial summer run of her talk show. It is set to air on Friday 23 August as the finale episode of Kris.
New mum Kim, 32, has already made a surprise appearance on her mum's show — albeit by video message.
"I just wanted to say congratulations on your new show," the reality star said on camera. "I'm sorry I couldn't be there in person, but I'm just loving life a little bit too much at home right now."
Fans are still waiting for their first glimpse of little North, who was born five weeks ahead of her due date, on 15 June at the Cedars-Sinai Medical Centre in LA.
The new family have started to emerge into the public eye. They were spotted out and about together for the first time on 15 August — although baby 'Nori' was carried in a car seat that was covered by a blanket.
The couple have received of plenty of media offers for the first pictures of their daughter, but HELLO! understands that they have so far not accepted any of these offers.