Shit, you think people feel guilty man? I don't really understand why, maybe the guilt should be eliminated rather than the attraction. Feelings aren't wrong, they're part of you, and guilt is as much a feeling as lust, both being rather absurd and complicated.
As I have commented before, I didn't feel guilt over my attraction towards young girls who I mostly saw as more beautiful in every way than adult females. I did feel some guilt I am sure when they entered my fantasy world that I created around their image perhaps. This is the reason I think we often do feel uncomfortable when looking at young girls, our thoughts have a tendency to race away into the fantasy realm.