How does having the wrong grip size affect you?

In my opinion and experience of over 50 years, I think too small is worse than too large. A grip that is too small will tend to twist in your hand when you hit an off center shot, and also that too small grip will make it hard to grip the racquet firmly when needed.