I guess it depends on your prescription? If you have 3.00+ eyesight like I did you wouldn't be able to see the ice that carefully, and it'll probably make a difference in your confidence to execute and stuff. Also I think it affects your depth perception so it's harder to see how far away the boards are. And....sometimes there are wet spots and holes in the ice you want to avoid. Although I agree, I played many years of tennis and your vision is everything! Sometimes even a bright glare or the angle of the sun affects your play.
As for falling and hitting their head, having worn contacts for 10+ years I've never had them fall out unless I was rubbing my eyes. If they fell out due to hitting your head you probably have more problems than losing your contacts :/