"I am not ready for that," his youngest daughter tells him in a preview of the next episode of E! reality series "Ashlee + Evan."
Joe Simpson was declared cancer free last year, but it wasn't an easy battle.
In Sunday's episode of "ASHLEE+EVAN," Ashlee Simpson-Ross learns her father almost considered giving up before undergoing surgery to beat prostate cancer after being diagnosed in the fall of 2016.
"I really wrestled with saying, 'I'll see you guys later,'" he tells his daughter, who gets emotional in the clip upon hearing the admission.
"Definitely not that time. I am not ready for that," she says with tears in her eyes. "You're so strong. I'm just so happy you're here."
He is, too, and says he decided he couldn't leave Ashlee or her older sister Jessica Simpson behind.
"I remember saying to the doctor, 'What if I don't have surgery?' He said, 'In six months you'll be dead," he explains. "I couldn't do that to you and to your sister and to my little grandbabies."
He may be cancer free, but says he's not 100 percent healthy yet.
"The biggest part is, because I don’t have testosterone, I just don't have energy," Joe says. "So you have to power through."
"Ashlee + Evan" airs Sunday nights at 10 p.m. ET on E!
Got a story or a tip for us? Email TooFab editors at firstname.lastname@example.org.