Is this some kind of a joke or something?
You said that you don’t want to wreck this man’s marriage, and that you thought you were only going to sleep with him a couple of times. The minute you let this man’s Willy into your House of V, you wrecked their marriage. And don’t let yourself fool yourself into believing that “It just happened.” Unless you were drugged or unconscious, and in that case it would be rape, both of you went into this unfortunate arrangement knowing exactly what it would entail. It’s like pricking a pin into a balloon, it only takes one prick, not a thousand, to pop the balloon.
Unless the man and his wife and you (all three of you, not just him) have agreed to be in an open marriage, then you know and he knows (and maybe his wife knows, but probably not) that this situation is wrong.
He can, indeed, end his marriage. All he has to do is file for divorce, then you two lovebirds wait until the divorce is final, and then re-connect. He has already told you that he doesn’t want to get a divorce.
He said, “He says I think of you all the time.” What does that mean? It means that he probably still loves his wife (or her money, or the fact that she most likely cleans his house and takes care of their kids) and that he like f*cking you. His wife either doesn’t know that he’s having an affair, or she doesn’t know what to do, to get him to stop. You’re not putting up much of a fuss, you just come a f*cking whenever he calls you to him, so this guy has a great situation for himself. If you don’t mind that you are, indeed, ruining his wife’s marriage, then keep on f*ckin’, oops, I mean truckin’!
If you want to have some semblance of doing the “right thing” (that has different meanings to different people) then you let Casanova know that you don’t wish to wreck his marriage and you either break it off immediately, or you let him know that you expect him to start divorce proceedings immediately (then wait to continue your “relationship” until after his divorce is final, even if it takes a year or more, because if he really wants you, he can wait, and so can you).
In the meantime, buy yourself a vibrator.