Should I end it with him?

I have a close friend who lives far away. We’ll call him James. We used to like each other but decided it’s best that we stay friends due long distance and preventing hurting a mutual friend who he has history with. Fast forward a few months, I started talking to someone who is perfect in every way. We’ll call him Matthew. He treats me incredibly well, and is perfect in every way. Even better than when James and I used to talk. James is also talking to someone new. Sometimes James flies here every few months and we hang out with our mutual friends. However, when James and I are alone, he’s very affectionate with me and gives me kisses on the forehead or hand. I’m not affectionate back, but I can’t repress the old feelings that sprout when he does things like that. But all I feel is guilt because it’s like I’m cheating on Matthew (who already doesn’t like James).

What should I do? James and I are incredibly close and I know that if him and I do pursue a relationship, it wouldn’t work out especially because of the long-distance. And I don’t want to hurt Matthew, who could potentially be a really good and healthy relationship.