Puberty has quite a bit to do with it. Imprinting at that time.
It happened to me. I was imprinted with an unusual orientation that happens to involve little girls. I do not want sexual intercourse with them. My imprinting is very much counter to procreation. Yet, here I am.
I suspect that this is true for all human sexuality. It's just a matter of what the duckling sees when it hatches: a momma duck or a lawnmower. Of course, I can't prove it. But it is what I suspect.
Oh, and before I press the "post" button...