I enjoy imagining that aspect of a relationship with really young girls, like under 8 maybe, being in the "caretaker" role. But for me, personally, I see that mostly to completely separate from any sexual attraction. A good deal of men I think, that are not pedophiles, would share the same feelings that I have.
For me, I first realized my attraction when I was 13, and it wasn't precisely the sexual attraction that started it, but I first noticed that there was something "special" about girls, and that even if they were my own age there was an obvious implication of my preference for them over women.
Not that this says anything about your attraction or experience - just mentioning my own.