Just because an infant cannot speak the same language as the older people around him or her does not mean, in my opinion, that the infant cannot communicate consent or dissent. They speak a language of sounds, body movement, and (albeit later) facial expressions. I would look to those for drawing a line because many sounds, body movements and facial expressions are universally understood to mean the same thing so far as I know. If baby is happy then let him or her be happy, if unhappy well then don't do that.

I'm speaking hypothetically of course, just thought I'd mention it though.