Of course, the key here is how secure the wife feels in the marital relationship. If she feels insecure, then there is a serious problem that you both need to deal with. In that case, whatever you do on the side with whomever pales in significance.
Adultery is just a word. Essentially, we are all adulterers. Any time that I have lustful feelings towards anyone not my committed partner, I commit adultery. The very way that humans are wired as sexual beings, I cannot see adultery being avoidable. No, the real issue on the table is not adultery, but betrayal.
Only if you feel that your are betraying your wife by spending time with your LGF, do you have a legitimate moral quandary. And that my friend, is something that you must resolve for yourself.