Holly Madison Details Sex with Hugh Hefner, How He 'Lashed Out' at Her & Tension at the Mansion

"Girls Next Door" was beloved by viewers who wanted a BTS look at life inside the Playboy Mansion -- but for the women living there, it wasn't quite the picture perfect situation it was made out to be on TV.

While Holly Madison already opened up about her experience as one of Hugh Hefner's girlfriends in the tell-all book "Down the Rabbit Hole: Curious Adventures and Cautionary Tales of a Former Playboy Bunny," she revisited some of the highs and lows from that time on this week's Call Her Daddy podcast.

Madison first explained how she wound up at the mansion in the first place, saying she scored an invite after working a gig for Hawaiian Tropic. As she started going to more parties, Hef's girlfriends invited her out with them for a night. At the time, she was "failing at everything" in Hollywood, had bad credit and was crashing with a friend. Drawn in by what she felt was a "welcoming" community, Holly also thought she might "have a safe place to live" for a bit if she took them up on their offer.

On their first night out, she said Hef tried offering her Quaaludes, which he referred to as "thigh openers." While Holly turned them down, she said she witnessed Hefner "hand them out for years" after she moved in. Madison said she would get really drunk and added that marijuana was popular in the house -- claiming she couldn't "remember a time in the bedroom I wasn't wasted."

"Everybody would go upstairs and basically everybody would have sex and I didn't know what to expect," she said of her first time having sex with Hef. "I think I had the feeling that, this is my first night out, maybe I won't be expected to do anything my first night. But then when I got up there, I was so wasted and thrown into it, that once it was done, there was something about the experience that made me feel very out of control."

"For me, I'm already in this desperate situation and I'd already thought living at the mansion would be a solution to my problems," she explained. "I think I felt if I were just to leave that night and never come back, I would have felt really used and really chewed up and spit out."

She said that women wouldn't be expected to have sex with Hef every night and would wear underwear to let him know when they were on their period. Madison said Hefner wouldn't ever "pleasure" the women and mainly laid on his back while they did most of the work. She also said Hef didn't pass the women around to his friends -- saying he was "always really careful about having other older men around" and was "very selective about how he curated his friends."

She said she had to sleep with Hef before being allowed to stay in the mansion -- a rite of passage all the women had to go through. "I'm not trying to slut shame or anything," she added, "but nobody ever got asked to move in unless they slept with him."

When Holly first moved in the mansion, there were six other women there at the time, including Hef's then-#1 girlfriend Tina Jordan. Madison said a number of the women were "dating other people on the side," something which made Hef "super jealous" when he found out. "I've seen people come in 5 minutes after 9pm curfew and get dressed down by him," she said, claiming one woman lied about visiting her family when "they were really f---ing Kid Rock or something."

Explaining the initial allure to Hef -- outside of the stability living in the mansion provided -- Holly said she fell for things she now knows "are super cliche of groomers," like him telling her she was "really special." She added, "It's hard to not move into that."

She admitted she didn't really know what she was signing up for when she moved in and none of the other women were all that helpful. "They would tell me the bare minimum," said Madison, "Nobody really wants to help anybody out. Everybody wants a centerfold or more money."

"It was horrendous the first three or four years I lived there. Before it was me, Bridget and Kendra, it was me and six other women and it would rotate every couple years. It was really cutthroat, nobody got along, everybody tried to snitch on the other," she recalled. While the other girls wanted to go out and party, Holly said she actually liked staying in for Hef's old movie nights -- and felt the women "pushed" her into the main girlfriend role when Tina moved out.

"Nobody wanted it," she said of the position, which meant she would have to move into Hef's room. "At first I was flattered by it ... I felt like I was really special. Meanwhile, everybody else had been there long enough to know what the deal was. Then everybody turned on me not long after because Hef realized he could use my good behavior as a comparison for the other girls. Why can't you just behave like Holly does? Then they hated me. They were like, let's get this bitch out of here."

She did eventually start to feel like she was in love with Hef, but added that, "looking back at it" now, "it was a very Stockholm syndrome kind of thing where I felt like I identified with him, he was complimenting me so much, I started in my mind started blaming all the other problems on the other women."

"After I had sex that first night, I had locked myself into the situation in a way because of the shame involved," she continued. "Imagine having sex with somebody in a room full of women who all hate you and you know they're all talking shit about you. Like, how horrible? I felt it the whole time ... yeah, that was kind of f---ed up. It was gross."

She went on to say she did have sex with Hef one-on-one after she moved into his bedroom, but was never somebody who actually felt sexual until after she left the mansion. Madison said she was always "too messed up" to orgasm and claimed Hef had "a fixation" on how the women all groomed themselves. "I feel like his obsession with young women was semi-gross," she said, adding that, "He would get mad if somebody wasn't shaved." When asked about his penis, Madison said it "was just like, normal."

Madison didn't actually say much about Kendra Wilkinson or Bridget Marquardt during the 90-minute interview, only mentioning them in passing while talking about the contracts they had to sign for "Girls Next Door." While she doesn't know whether Kendra -- who was out of town when the contracts were presented to them -- actually signed hers, she felt she and Bridget were "under duress" when they signed theirs. All three of them had reservations, she claimed, with Holly saying "it was creepy ... it felt like it was a contract to be in a relationship." Of course, they all did and the show became a massive hit.

When asked whether she had a relationship with Kendra at all now, Madison simply said "No," before adding that she and Bridget have "always been close."

Eventually, Madison and Hef started talking about having children, something which didn't happen and was the beginning of the end for her time in the mansion.

"I knew it was because of him, I was healthy, he was just too old. Thank god [it didn't happen]," she said, explaining that having children was another example of her trying to justify why she stayed there so long. "I have to make this situation into something, I'd almost locked myself into this box in a way. Which isn't hard to do there, it's a very cult-like atmosphere anyway and you're manipulated to feel that way, but on top of it, my own shame kept me there too. I just couldn't really imagine a life outside of there."

"I thought this is my last stop, if I want to have kids, I'm going to try," she added. "I was like, okay, if I'm not going to have kids here, this is something I need to think about. This is like a death sentence."

Madison claimed she felt suicidal at times in the mansion and was given anti-depressants right around the time the show began. As time went on, she said she noticed she started to develop a stammer and her "brain felt slower" the longer she stayed in the mansion. "I think it was part of being nervous, partly doubting myself, partly being in this environment where I was always drinking and my mind was never stimulated," she explained, "I was kind of always in fear of getting sabotaged by other women, getting kicked out, him yelling at me or embarrassing me in front of other people."

She then started to think about leaving, a decision which took her months to finally come to. She said she had "a lot of guilt" about leaving, because she felt like "I made a commitment to be with him and we talked about being together for the rest of his life. I felt a little guilty about leaving."

"But when I realized I wasn't going to be able to have kids with him, that was a big thing ... and my depression was coming back and he would go and ask Bridget why I was depressed and he wouldn't ask me," she alleged. "It was just really weird. As everybody was leaving, I started to realize it's just me and Hef, he started lashing out at me more. I don't know if he was just stressed ... I know things weren't so great financially for him, maybe he had some stress, or didn't have the games of pitting the three of us against each other anymore. He started lashing out for really stupid things and I just realized I can't be here. This guy is an asshole, I can't stay here."

She said it took her being interested in another man, Criss Angel -- who she only referred to as "He Who Will Not Be Named" -- to force her hand, because she didn't want to cheat on Hef.

"When I left the mansion, I got sucked into another relationship that was really bad," she continued. "I was so ready mentally to be independent, but it was another love bombing relationship and I just fell into that. I really feel like it was sent to me by the universe to be this final dot on the I, to be like, okay you need to never be in a relationship like this again."

She said other men in her life since Hef have been "really insecure" about her past. "Some have raked me over the coals for it and ask a million questions about the sex," she added, "Some guys are super insecure and can't deal."

Madison also said she didn't really have any regrets about her behavior on "Girls Next Door," because she sees certain things as coping mechanisms now.

"When I look back at Girls Next Door, especially the first few seasons, I see myself just coping. I was really stiff and not open and just like a robot and saying the type of things I felt Hef would want me to say and things that made him look good," she explained. "I can't really don't regret that because I don't know how else I would have coped. Now, I'm more myself and more open. I couldn't have said what I was really saying or feeling, it would have been cut. I was super guarded and super nervous about putting my private life out into the public."

Hefner died at the age of 91 in 2019, but did speak out against a number of Madison's claims in her tell-all book in 2015. "Over the course of my life I've had more than my fair share of romantic relationships with wonderful women," he told Us Magazine at the time. "Many moved on to live happy, healthy, and productive lives, and I'm pleased to say remain dear friends today. Sadly, there are a few who have chosen to rewrite history in an attempt to stay in the spotlight. I guess, as the old saying goes: You can't win 'em all!"

Click here to listen to the full interview.