It's just "wear armor while casting." I don't see a big problem, other than obviously you'll need a decent strength to wear even mithral full plate so this prevents you from dumping it entirely.
Remove Fear is a thing - potions, or Magical Lineage + Logical Spell etc. Once suppressed, Intimidate won't stack and so you will be immune to it for 10 minutes.
There are also plenty of spells with no somatic/emotion components, like Dimension Door, just learn one of those.