Troy - who is credited with the singer's meteoric rise to fame - has admitted that he wasn't prepared for the dismissal, which took place on the eve of the release of Lady Gaga's ARTPOP album in November.
"It's like you wake up and work with somebody every day, and then all of a sudden they're not there anymore," he told Fast Company.
"I don't think you're ever prepared to sever that deep of a relationship."
Recalling the first time he met the star, he said, "She walked in with these huge sunglasses on, fishnet stockings, and basically told me how she was going to change the game.
"She was a performance artist calling herself Lady Gaga, who had a European dance club sound and pop star aspirations - elements that historically haven't mixed. Def Jam Records had just dropped her. I believed her."
Meanwhile, Lady Gaga told fans she had been "gravely mismanaged" as she apologised for the delay for her new music video, Do What U Want, this month.
Troy had worked with Lady Gaga since 2007
"It is late because, just like with the Applause video unfortunately, I was given a week to plan and execute it," she posted on her Little Monsters website.
"It is very devastating for someone like me, I devote every moment of my life to creating fantasies for you. All my most successful videos were planned over a period of time when I was rested and my creativity was honoured.
"Those who have betrayed me gravely mismanaged my time and health and left me on my own to damage control any problems that ensued as a result.
"Millions of dollars are not enough for some people. They want billions. Then they need trillions. I was not enough for some people. They wanted more..."