Andy Cohen Reveals Taylor Swift/Katy Perry Story Taylor Didn't Want Anyone to Know About

Andy Cohen just released a new memoir, "Superficial: More Adventures from the Andy Cohen Diaries," in which he -- among other things -- details an interaction with the "Shake It Off" singer about Katy Perry she didn't want anyone to ever know about.

It's no surprise the two have a lot of "Bad Blood" between them, so much so Swifty wrote a song about it ... but a conversation between Andy and Taylor at the Met Gala shows how deep that battle goes.

"So the Taylor Swift thing is, I saw her at the Met Ball after-party, I don't know her and I have a history of saying exactly the wrong things to people and I did with her," he explains in a new interview with PEOPLE.

"I had seen Katy Perry and I was standing there waiting for a drink at the bar and Taylor Swift is walking by saying to her friend, 'Where can we go stand, it's so crowded here and Lady Gaga's about to sing' and I turned to her -- she wasn't talking to me, but I inserted myself -- and I said, 'Your friend Katy is over there and there's no one sitting over there with her.'"

Taylor's reaction: "Katy Who?"

After Cohen told her he meant Perry, he says Swift "basically made it clear to me that this was the exact opposite of her friend and then kind of [said] don't mention it on my show, in harsher words."

"She kind of came back and put her hand on my shoulder," he adds. "I had said to her after, 'Oh my god, I am so stupid, I am so sorry.' I thought wow, she thinks I'm being shady."

"I said to her, 'At least you know where not to sit,'" he explains, as he tried to lighten the mood, "at which point she said 'Maybe don't mention this on your show.'"

While Andy says he has talked about the moment on his Bravo series, it's only starting to make news now with the release of "Superficial."

"I maybe am a shady ass bitch that I did put it in the book," he admits, saying he thinks the real purpose of the story is to show "what a douche I am in that instance."