Burn them with fire wait, wrong window.

Ahem.

Zincorium speaks truth. Basically, you're organizing a regular social event, which you host, centred around an activity. If some of the invitees are disruptive enough that the rest of the group can't enjoy the activity, the only real options are:
a) ask them to behave better, making it clear that the issue is the group's enjoyment, not "I have control issues" or "I am a buzzkill" or any other defensive reaction they may have (and of course, be sure that that's true!);
b) if that doesn't work, disinvite them.

It's a matter of social protocol and not really something that can be solved through the game or rules around the game.