I'm curious as to how you'd relate this to minorities such as:

Deaf
Blind
Physically challenged
Mentally challenged
Elderly

That's a rather asinine comparison. Those are all people with disabilities and an age group which is more heavily associated with physical/mental impairment than all other adult age groups. A much better comparison would be with racial, religious, and sexual minority groups such as blacks, Jews, gays, etc. There is nothing inherently disabling about having an age based chronophilia such as pedophilia or hebephilia. It can only be socially crippling due to social unacceptability. But were these things to become socially acceptable, it would cease to be an issue. So your premise is flawed here.