I agree with your basic point.

I rarely people who make this argument being as specific as to say "not attracted because of their appearance".

They have no clue what it feels like to be attracted to kids. A common understanding is that men who abuse adult woman enjoy the power component. When it comes to child sex abusers, where the potential power difference is large, they will assume that is the motivation. It's something bad they can think about pedophiles, and they are ideologically inclined to find as many bad attributes as they can. Indeed, child sex abusers will make use of power to enforce keeping of a secret.

I think that's an adequate explanation for their views, without needing an entire ideology such as Dante provides.

In reality, I suspect it's rare that abusing pedophiles actually wanted power for its own sake even if they use it to enforce secrecy. Celibate pedophiles if anything seem to shy away from being assertive with kids even when it is appropriate (like to keep the kids from mistreating them).

Having read extensively about Michael Jackson (in Tom O'Carroll's book) and some snippets from Jerry Sandusky, I saw no hint of either one enjoying power over boys even as they did bad things.

In my blog and media interviews I will sometimes challenge this idea that pedophiles like the idea of controlling (let alone damaging) children.