There are two versions of each character. The one in the player's head and the one in the GM's head. When they get out of sync, someone has a bad time.

What I do when a player attempts something that seems out of character to me isn't to tell them "so-and-so wouldn't do that, so you can't do that." Instead I use something like "That action doesn't jive with my understanding of so-and-so. Could you tell explain why it's in character so I can fix my understanding of who the character is?" It's totally non-confrontational. The player can back out (they usually do), but it doesn't look like I took their choices away. I'm sure an antogonistic player could abuse me, but I try to avoid inviting them to games.