If "a humanoid in armor wielding a sword" is a blank slate that you can't project anything practical onto, I'm hard pressed to see how "a humanoid in robes with a spell component pouch" is much different. You might normally expect the latter to be vulnerable to Counterspell (which has been discussed as a problem spell) and would let PCs with racial magic resistance have an edge on saves (something I just touched on, and that has support even from a lot of the "most of the changes are okay" crowd), but their spell selection would be a blank slate outside of maybe expecting wizard staples like Fireball. Outside of environmental details or environmental queues, most humanoid enemies are hard to pin down exactly beyond a very small number of archetypes. Even if your DM could stand to be a little more evocative.

Assume we all agreed to ignore Counterspell as being problematic, maybe worked around Globe of Invulnerability and Antimagic Field as similarly prone to shutting down whole categories of enemies, and then let all the other antispell effects that weren't hard counters take effect against nonspells. What else would you want done differently to make it feel less "calvinball"?