The 47-year-old actor, who entered rehab after a liaison while filming in Canada last December, was speaking during the filming of her new reality show True Tori.
"We have four kids, so.... there were ebbs and flows," he explained, asking his wife: "Is that safe to say?"
The Hollywood socialite didn't agree, replying: "We had a great relationship and we had a great sex life."
Tori's husband said he wasn't getting enough attention at home
Dean told their counsellor that they were intimate only once every two weeks and this "wasn't fantastic" for him. Taking 40-year-old Tori's side the therapist said his view of marriage is "like a fairytale".
As the discussion unfolded it all became too much for the mother of four who burst into tears - prompting Dean to reach over and comfort her. "He's never going to be happy with just me," she wept.
Married for eight years, the couple are parents to Liam, six, Stella, five, Hattie, two, and 17-month-old Finn.
Dean was unfaithful while on location in Canada
Tori's husband described his betrayal of her was his "worst nightmare". "I cheated on my wife. That's my worst nightmare." She responded that perhaps it was just a nightmare because "you could caught". He didn't have an answer for to this.
The couple have agreed to film six episodes, with the series being shown from 22 April in the US - only three weeks after filming started.
The Lifetime network said: "Since viewers will be seeing events almost immediately after they occur, nobody knows how the drama will unfold — not Tori, not Dean — and only time will tell if Tori and Dean's marriage survives."
Tori has said the docu-series is a chance to tell her own story
Talking about her controversial decision to film this difficult stage in her life, the reality star said the programme was her "chance to tell my own story" and "get my voice back". In the blog post she thanked fans for their "unconditional love, positivity and support".