Dealing with Genre Disconnect

[Justin_Bacon]

I do my best to hint to them what kind of story they're in.

Emphasis added.

That's your problem in a nutshell. Don't try to hint at this. You should just flat-out tell them what type of campaign you're aiming for. Either they'll want to help you achieve the atmosphere you want (in which case your problem is solved) or they won't want to play in that type of campaign (in which case you're not going to solve the problem by beating around the bush, either).

My guess is that you're giving them a situation X and thinking, "X should give them a good opportunity for roleplaying dread and fear."

But when they're seeing X they're thinking, "X is giving us a good opportunity to roleplaying how badass we are in overcoming our fear and dread with a stiff upper lip."

So I'm thinking of going back to using Sanity Checks (the optional rule from Unearthed Arcana, made famous by the Call of Cthulhu RPG). I tried using them in the first campaign I ever DMed (2005), but ultimately abandoned them because players kept passing their sanity checks. With flying colors. I had a solid group of bastards so unshakable it became silly, and I decided not to use Sanity Points again.

Sanity points only work insofar as the players are willing to use them as guidelines for roleplaying. (However, using them is one way of sending a clearer message about what type of atmosphere you're looking for.)

Personally, I've always preferred the sanity system in Unknown Armies. It's a bit richer, more responsive, and more personalized.

In my current campaign I'm using a variant of the Taint rules (also from UA). This has the nice effect of actually terrifying the players. (Although I suspect that will fade once they get access to the spells that will fully purify the taint.)

I'd actually caution against talking to them directly. Asking them to play their characters as "scared" is undue influence and probably going to be resented even if it goes through.

There is a difference between saying "this is what your character should do" and saying "here's what the campaign is about".

Open communication is, frankly, never a bad idea.

IMO, you're better off trying to use some of the flexibility you have as DM to induce that state in players and characters via more subtle means. If you think the Fighter should be getting scared, roll DC will saves vs minor phobias

This I would find infinitely more annoying as a player. In fact, I find it kind of silly to say "players will resent you for telling them what genre you're trying to achieve, but they'll be A-OK with you making up rules out of whole-cloth to force their PCs to behave the way you want them to".