The "you just have to wear it, it's easy" crowd doesn't seem to appreciate the type of movements and precision you would need, presumably, to fight supernatural monsters and dodge magic. It seems like a failure to even try imagining that wearing armor properly and efficiently in D&D might mean something a little different than just comparing it to the real world.

In the same way I couldn't realistically kill a D&D dragon with a longsword if it lied on its back and let me coup de grace it in the chest over where its heart would be, the game still allows me to hack at a dragon with a longsword enough times that it would die.

Similarly, it may seem like "oh armor is just like clothing, why wouldn't someone be able to just wear it?", but the game is telling us that only people trained specifically in its use can wear it well enough that it doesn't interfere with their actions.