"I just think people come into our lives and think it's maybe easier than they think -- like to be with you or to be with me," Scott replied. "It's like very true that we do come with a lot of baggage and it's definitely not easy that we see each other, work together and are friends."
The Talentless designer continued to point out how complicated the situation was, as the former couple is very much involved with co-parenting their children Mason 11, Penelope, 8, and Reign, 6.
"I don't think I can do it the other way around, but I've always been clear that my priority has been my children, my life with them," he explained. "And I even put it out there that taking care of you is one of my priorities."
In a confessional, Scott gave props to Sofia for being a "trooper," but admitted it must be hard for anybody to see the person they are dating hanging out with their ex.
But then he turned his attention back to the mother of his children.
"I just don't know, like, are we just going to like grow old, just traveling the world with the kids, living like one house down from each other or -- together at some point."
Watch Kourtney's reaction in the clip, above.
The final season of "Keeping Up With the Kardashians" premieres Thursday, March 18 at 8:00pm ET/PT on E!.