D&D adventure design: Why can't campaigns last to level 20 and beyond?

[Azuresun]

I really don't understand the position that there is something wrong with having defenses in place to counter the expected uses of spells. Sure, Fly can get you to the top of the tower, but in a world where Fly exists the top of a tower occupied by a relatively successful bad guy should be either defended or inaccessible. That's not going out of your way to counter a spell, that's simply treating your bad guys with a little respect. It's not like the party are the only people in the setting who are aware of the existence of spells. And if the bad guys are aware of the existence of spells, they should be expected to have reasonable counter measures in place.

The problem is that either they can use the spell to undramatically sidestep the interesting bits, or it gets countered and it turns out they can use those spells, just not to do anything meaningful.

And for "Fly", you could sub in Commune, Scrying, Speak With Dead, Teleport....it's hard to find the narrow band where the PC's get to feel cool for using those abilities in a meaningful way, but not feel bored because they're using them to sidestep all the interesting bits of the story.

[OldTrees1]

The problem is that either they can use the spell to undramatically sidestep the interesting bits, or it gets countered and it turns out they can use those spells, just not to do anything meaningful.

And for "Fly", you could sub in Commune, Scrying, Speak With Dead, Teleport....it's hard to find the narrow band where the PC's get to feel cool for using those abilities in a meaningful way, but not feel bored because they're using them to sidestep all the interesting bits of the story.

I am not sure the band is that narrow. Although I understand the concern.

If I were defending a tower in a world where Fly was a 3rd level spell and Teleport without Error was a 7th level spell, this is how I would lay out the tower and my defenses.

1) I would consider the "innermost" part of the tower to be the 2nd basement and 3/4ths of the way to the top of the tower.
1A) I would split interesting things (whatever is being guarded) around those points if possible. However some things cannot be moved that far. A beacon for example might still be on the top floor and nobles tend to refuse to live in the basement.
1B) I would lay out guards in rings around the surface of those important areas, more details in #2.

2) I would consider the surface of the tower to be any entry point, but weighted based upon likelihood that entry point would be used.
2A) Fly means the top of the tower is an entry point, but there are more non fliers than fliers. The same goes for climbing to the windows. So some guards would be at windows / the top of the tower but less than elsewhere.
2B) Teleport means every space inside the tower is an entry point, but there are more fliers than teleporters. So some guards would be everywhere inside but much fewer than even at the top of the tower.

5 guards at the main door.
2 guards for the top of the tower and other major windows. (Let's assume there are no other major windows)
4 guards that patrol the tower in pairs.
1 guard in each of the innermost parts of the tower.

As a result things like flying mostly sidestep the defenses but not completely. Flying was not made meaningless nor were the guards idiots.
Ground assault: Take out 5+2+1 guards
Flying assault: Take out 2+2+1 guards
Teleport assault: Take out 1 guard.

[MirrorDarkly]

It's interesting how different my experience with "problem spells" is as a GM. I run homebrew and tend to structure my worlds so that abilities like Fly etc are what allow the PC to access the next tier of content, so I rarely feel like they are a problem I have to work around.

For example in the game I'm currently running session 1 the boat the players were on was attacks and sunk by an airship. They fought off some of the initial raiders it dropped but in the end had to decide what NPCs to save (which gave them allies, enemies and various deferent story hooks depending on who lived and died) as they ran for the lifeboats because the airship itself was impossible for them to meaningful harm.

Now they are level 8 and have access to fly, dimension door, etc and they have decided they want to steal that airship, and I've created a set of adventures around that goal. Zone of Truth has been used to root out the traitor who betrayed them in the beginning, but only after some legwork narrowed down the suspect pool, and they are going to use him to lay a trap for the airship. They've sailed the River Styx to recruit a Merrenoloth to captain it, etc...

Sure they can bypass or stomp all over a tower full of mundane guards now but that means they get to deal with the faction behind that tower in one way or another instead of just not having anything fun to do for the night. Also I'd don't believe having counters for PC abilities is a problem as long as they are not always countered. Of course you are not going to be able to just teleport into the inner sanctum of the guild of super mages or whatever anymore than you'd expect the rouge's guild to have only a simple DC 10 lock on it's main door. But the PCs can use those abilities and spells to gain access to things that will eventually get them to those places.

In fact I had a lot of trouble running 4e because it PC ability curve tended to be just getting higher numbers instead of new ways to interact with the world. It made it very hard for me to "section" my world in a way that they would naturally gain access to the appropriate parts of it as their levels increased.