You mean like how D&D has special worlds that are full of hostile environments and powerful natives? The idea that it would make absolutely no sense for people to mostly ignore parts of the world that are incapable of producing anything they care about, either in terms of resources or threats, is frankly bizarre. That's exactly the way people behave historically!
There's always going to be a point between "a threat is worth wiping out" and "that threat is actually wiped out", and a mechanism by which that threat gets wiped out. It's not really unreasonable for "adventurers" to fit in there, especially for things less mundane than "a tribe of goblins". If the problem is that a cult has set up shop in the woods outside town, it's pretty easy to understand why that won't necessarily correlate with existing borders. Nurgle (or Orcus or Ruin or Cthulhu or whoever goes around setting up cults in your world) doesn't have a material kingdom he sends out envoys from, he just shows up in people's dreams and tempts them with promises of power.