"When he passed away, everybody expected me to have some big reaction or post about it on social media, and it just felt weird," Madison said. "I didn't have any emotional attachment to him anymore in any way."
The Girls Next Door alum dated Hefner for seven years, from 2001 to 2008. She has been very outspoken about her time living in the Playboy mansion, and relationship with Hefner, sharing her experience for the first time in her 2016 tell-all book, Down the Rabbit Hole: Curious Adventures and Cautionary Tales of a Former Playboy Bunny, which was released a year before Hefner's passing.
"I'd already come out talking about what a toxic relationship this was for me. Why am I supposed to post a memorial on my Instagram?" she told PEOPLE, admitting that she didn't feel "relief at all."
"I felt like I had taken myself kind of out of that universe pretty solidly. But it was a really odd time," Madison continued. "For me, after leaving that relationship, I kind of felt like he had always interacted with me in such a fake way. Because every interaction he had with me was all about control or this fantasy he had of a relationship. It almost felt like playing house in a way."
After the podcast host split from Hefner and left the Playboy Mansion, she also exited The Girls Next Door reality series. Madison went on to star in a spinoff series, Holly's World, from 2009 to 2011, which also ran on E!. While speaking with PEOPLE, she claimed the producer of Holly's World continuously tried to get her to talk to Hefner.
"The spinoff was produced by the same guy who did Girls Next Door, so he would want to do crossover episodes," Madison said. "So I did a few of those and it felt safe because the cameras were on, and it's semi-scripted, so that felt okay, but if I was in a meeting with him in the office, he would always be like, 'Why don't I get Hef on the phone? Why don't you guys talk?''
"Why would I want to talk to a robot? Everything he says to me is just going to be designed to get a certain reaction, designed to keep me on his good side. It just didn't feel authentic," she added. "Before he passed away, there had been maybe five or six years where I just had not spoken to him at all. He had become a completely different character in my mind."
Meanwhile, although her book came out over seven years ago, Madison said she still receives criticism about it from "snarky" people to this day.
"There's still people who want to attack me for telling my truth, even if it has nothing to do with them," she told PEOPLE. "Just because when you're coming from the mansion, it's kind of this high-stakes environment where people attach who they are to this title of being a playmate or being associated with Playboy, so they don't ever want to hear a bad word about it."
She added, "And it just gets really catty even now with some people, but those people aren't in my life, so it's okay."