One very important factor is are we including magical beasts into the equation? A single Dementor could theoretically kill an entire community while invisible and I'm not even sure if they can be hurt through physical means. In the movies maybe, but in the books they are kind of insubstantial, more of a presence than anything else. We also hear that giants can create effects not dissimilar to tornadoes, so without too many giants, and there are a bunch, civilian centers would take massive damage.

Even without the creatures they do still have mind control and teleportation, so with efficient use of both hey could control most governments within a year. As for whether that happens, it doesn't with Voldemort. He would hold the muggles in contempt and go for a very aggressive campaign which would kill a hell of a lot of muggles, but a lot of wizards as well so he'd be fighting a losing war, mainly by numbers. The wizard population would not be wiped out because of all the safe havens and ability to be unchartable, but he personally would not win a war. Dolohov might be able to.