I heard something similar

I actually had a therapist say the same thing pretty much. He just said there's nothing wrong with being attracted to young girls, it's common and normal among men. The only interest is how people deal with that attraction.

He gave an example along the lines of you can stop at a crossing and see a young girl crossing and find her attractive and that's fine. If you whistle at her or try to do more than admire from afar then it's an issue.

I don't fully agree with him but I understand his point, at least in the society we live within. Society has decided on a set of social norms and whether they're good or not we have to exist within them.