Railroad Bad, Trolly Tracks Good

[Duff]

There's a Lamentation of the Flame Princess module, God that Crawls, that illustrates why you'd use random elements even with a main antagonist. Simply, it has a monster that chases the characters through a fairly intricate maze. The module proposes two models for implenting this: the hard and the easy way. The hard way is that you track the exact location of the monster and each character individually, using by-the-book movement rates. The easy way is that there's a random chance for the monster to appear from a viable direction, which goes up with character actions that'd draw attention.

Both obviously work, but the first requires several on-the-spot calculations each turn, while the second gets away with one or two. A more universal example would be weather: you could plot a calendar for your campaign, deciding beforehand which days it rains. Or you could roll a die when the characters go outside. Both work, one is less work, you get most of the same results either way.

Much depends on which game you are playing.

In the Old School Essentials (D&D B/X) campaign I'm running, random encounters are an essential part of the mechanics of the game.

They are a cornerstone of the resource management puzzle for an adventuring party: do you spend another turn searching for a suspected secret door, risking more interruption by wandering monsters, or do you press on to clear more rooms before you run out of torches?

You're both right here - these are good examples where random encounters do have value. In these cases though, the table is specific to what is in effect one long scene. In the first, it's part of what is effectively a minigame of maze running.
In the 2nd it adds suspense to the search - To me an important difference is that the entries on the table won't be "An Owlbear". It'll be "The Owlbear from room Q is looking for food and sweeps through the area the players occupy". It wont be "2d10 Goblins" It'll be "A patrol of 5 goblin warriors led by a 2nd level Goblin fighter. As soon as they see the PCs, 2 warriors will try to run away and alert others while the others try to keep the party there".