I think I'd have to side with the people who can teleport, turn invisible, mind control and loook like anyone. Even before you get into the really insane stuff like time travel and the ability to hide things so that they're literally impossible to find unless the guy hiding it wants you to find it.

If Voldemort just continued his tactic of covertly mind-controlling or replaceing everyone in a position of power then he's win before anyone realised what was going on. Even if somebody did work it out what could they do? Litterally anyone could be an aganet working for the wizards and they might not even know it themselves so you can't truat anyone else, if you you try to confront them about it then they can just wipe your memory or mind control you, you can't try and capture them and interrogate them because they can just teleport out. At most you could try and get the drop on them and shoot them before they react, but then all you've done is kill one agent and probably volunteered yourself as the next agent.