"You've made an incredible assumption about me," Hoffman tells the "Last Week Tonight" host.
Last month, Hoffman was accused of groping the now chief coach and co-founder at ArcVida, Anna Graham Hunter, when she was a 17-year-old intern on the set of the actor's 1985 TV movie "Death of A Salesman."
"This is something we're going to have to talk about because... it's hanging in the air," Oliver said in the middle of the film's anniversary panel, which also included co-star Robert De Niro as well as the "Wag the Dog" producer and director. The Washington Post was there to cover the unexpected confrontation.
"It's hanging in the air?" Hoffman replied. "From a few things you've read you've made an incredible assumption about me. You've made the case better than anyone else can. I'm guilty."
According to the Post, the 80-year-old actor proceeded to reinforce that he does not remember meeting Graham Hunter nor does he remember groping anyone. In the video of the conversation below, Hoffman does appear to defend making sexualized jokes on set with the cast and crew.
"I still don't know who this woman is," Hoffman said. "I never met her. If I met her it was in concert with other people."
Oliver found Hoffman's response rather interesting considering he previously released a comment stating his behavior on that set was "not reflective of" who he is.
"It's that kind of response to this stuff that pisses me off," Oliver said. "It is reflective of who you were. If you've given no evidence to show it didn't, there was a period of time for a while when you were a creeper around women. It feels like a cop-out to say 'it wasn't me.' Do you understand how that feels like a dismissal?"
Despite the other members on the panel's attempt to move the conversation forward and focus on the film they were all there to discuss, Hoffman circled back to the topic at hand, accusing Oliver of having a closed mind.
"I would not have made that movie if I didn't have an incredible respect for women," Hoffman said. "The theme of the movie is he became a better man by having been a woman."
"I can't leave certain things unaddressed," Oliver said wrapping up. "The easy way is not to bring anything up. Unfortunately that leaves me at home later at night hating myself."