That rule is for dispelling an actual +X magic sword. Not a sword with a free-standing GMW buff on it. Come on mang, D&D 101.
So because you prepared for something, the GM is forbidden from countering it? Even in a world where enemy spellcasters know that magic buffs exist and can cause them problems if left alone? Where many monsters and nearly every class gets dispel magic as a spell or SLA? Doesn't that strike you as boring?
Magi can cast in heavy armor just fine.