This. Honestly, my problem is that there are some viewpoints that, directly or indirectly, harm others. This can be as dire as the example I gave above, or as minor as the fact that I know people who won't go to the gym because they're terrified that using the correct changing room would cause them to get a terrible reputation.
The problem is, as you have said, power without accountability. I don't want to go into details, but you can in theory set up a system where the censors are held accountable by their audience. But that's hard.
To give an example, The Playground actively moderates topics of discussion, as do other forums, but I've seen it abused in other places much more than I have here. I suppose it's either good luck as to who got picked as a moderator, or increased accountability, but at the same time the places on the internet I've seen with no moderation ended up becoming more echo-chamberesque due to either design or a generally off putting attitude (although this may be just my experience).