I think that the understanding that different people have different mental maps is one of the key skills to GMing, and probably one of the key social skills overall.
Making sure your mental maps are aligned is one of the best things you can do to make your game run smoothly. And it's not entirely on the GM, but it's heavily on the GM as they are the ones that know when actions are incongruent with their mental map, and it's their mental map that is authoritative.
Your characters may be "fragile" in ways that you simply know about and know to avoid. Most games I've been in wouldn't think twice about a brothel.
That's not saying that your table is wrong in any way - just that because you've played together for so long, you know where their landmines are, and so it's easy to avoid them.
That's a lot harder with people you don't know, especially if they have different moral backgrounds.