"We were negotiating a deal with her and we couldn't reach an agreement on the deal," he said in an interview with PEOPLE TV's Reality Check.
"I'm just sad we couldn't reach an agreement for next season," he added. "I'm kind of living in that sadness."
Richards has yet to speak out about her departure, with her rep simply confirming she was leaving the show. For regular "RHOBH" viewers, it's pretty clear why she left.
After a great first season, one in which she made headlines for how candid she was about her sex life, Richards and the women just never got on the same page this year. They butted heads almost the entire time, as her longtime friendship with Lisa Rinna totally fell apart.
After Brandi Glanville popped up with allegations they hooked up -- claims which were eventually denied by Denise and repeated ad nauseam -- the writing was on the wall.
"She's answering her truth," said Andy. "She has a family and kids. Whatever happened or didn't happen, I would imagine that it's something that maybe she didn't really care to discuss on a television show."
The final episode of this season's reunion airs Wednesday on Bravo.