sounds like that is the only way to go forward in your present circumstances
Good Luck but That is exactly why one should never "come out" if you want to or intend to have little friends. you are basically trying to roll back peoples perceptions of you after you made a big deal about your sexuality.
If you (like me) never had any adult relationships and never have given any impressions that you are interested in sex, people who know you, and like you, are free to assume your "good" (from their perspective) intentions.
Some "pedo" who is married and/or always running around with various women falls right in to the stereotype of the molester next door. However a nice guy who admits nothing can be assumed to be an asexual harmless man-child.
If you never "come out" people cannot reconcile the idea that a nice guy who all the girls love could possibly be a "pedophile".
and yes, I walk the walk, LOL, I sweet talk, make googely eyes and hold hands with girls in front of adults without being banned or blacklisted, and I'm allowed to be with them in private also.