I could not have asked for better responses to prove my point. It all led to this. Perfect.
Removing player agency, and reducing player agency, 'is totally fine', if it makes sense within the story. Having fixed locations, with fixed scenarios, with tough challenges that happen to the players, 'is totally fine' so long as it makes sense within the story.
Removing and/or reducing player agency is just something DMs do. All the time. Ideally, they do so because of a reason that the player(s) are willing to accept; Either a) The actual rules of the game, or b) some kind of narrative setup (e.g; Paranoid Wizard does Paranoid Wizard-things). Players are forced to accept that the DM controls pretty much everything, and it's up to the DM to make that control as palatable as possible.
You're walking down the hallway, and in your path is a 10ft. wide hole. On the otherside is a ladder. Judging by the markings in the floor, you can see that the ladder is pushed over the hole, and the Goblins clamber across it, and pull it back after them once they're across. How do you want to proceed?
'Can I jump the 10 ft. hole? I have a Strength score of 18 (+4) and Proficiency in Athletics.'
No.
'Why not?'
You can't jump the 10 ft. hole.
Almost no player will accept that. If they do, they'll come to the internet after-session and complain.
This tunnel was built by Goblins. It's cramped, it's stuffy. Medium creatures over 5 ft. tall are moving as though in Difficult Terrain. You see a 10 ft.-wide hole in the floor, with a ladder on the other side. What do?
'Can I jump the hole? I have big strength, and proficiency in Jump.'
No.
'Why not?'
Well like I said, you're over 5 ft. tall. You're walking in cramped conditions as it is. You're not going to be able to have the vertical movement to get any sort of clearance. And of course you can barely take a runup in the cramped quarters. So yeah, it's not really possible to Jump in this specific scenario.
'Oh that makes sense. Can I climb down the hole, and then climb back up the other side?'
Same deal. Both DMs are removing the ability to Jump over the hole. But one DM actually has a reason for doing what they're doing, and no-one is going to complain, even though in-game the exact same thing is happening. It's the way the DM presents their authorial control, that matters.