The duo sparked the budding love story when they were spotted holding hands at Nobu Newport Beach earlier this month; TMZ also posted a picture of the pair enjoying a meal together on another outing at The Ivy in Santa Monica.
But according to Fitzpatrick, people are just making assumptions because "I think you guys just saw us at lunch once."
The 30-year-old insisted he has never even discussed romantic involvement with the "Bachelor In Paradise" star.
As for whether or not he has any romantic interest in her at all himself? "Listen, Amanda and I are just friends," he said.