Game Masters taking control of player characters

[Pyrous]

That this is true in the general case is self evident.

What about when the Player tries to do some thing more like this: "Spell XYZ is banned in this campaign" "My Wizard researches spell XYZ and then casts spell XYZ". Is this a case where the action should be retconned/countermanded/otherwise not happen?

Yes.

"Oh, your wizard researched spell XYZ, that I told you was banned. I guess that's my wizard now."

"Spell XYZ fizzles."

"Please don't do this. We agreed that this spell was banned."

If the answer is yes, then how does one identify the divide between "retcon/countermand" and "now they are an NPC"?

If the group agreed to a houserule, and one player breaks it, they are giving up the PC, trying to be disruptive, or simply forgot/weren't paying atention. The first results in NPCing the character, the last in retcon/countermand, and the second needs to be dealt OOC.

Of course, the GM should always remember the player that there is an agreement to not take the action. If the player says "My character would do this and if he gets NPC'd, that's OK.", then you should NPC the character. If the player says "Oops sorry, I forgot.", you retcon it.

And if the player says "It's my character and the rules says I can do it and you can't take my character.", then the player is being deliberately disruptive. You should stop the session and discuss this with the group.