This is the explanation that makes the most sense, I think, with what we see in the films (original, prequel, and sequel). Swinging something and keeping it away from one's own body is well within the bounds of normal human capability, but without the Force, you won't be able to do the specific things the Jedi do with a lightsaber and will be stuck holding a glowing melee-only weapon in a setting replete with ranged weapons. One could argue for the usefulness of lightsabers in utility contexts for common soldiers, but at that point, it becomes a question of utility versus cost and availability.