If you get totally stuck. You could roll a die to dictate your mood (and hence give a bit of a nudge to your improvisation and eyeballing), 1&2 you are harsher that you initially planned to be, 3&4 you go with your common sense , 5&6 you are more generous.
It would break you from just enforcing your will, while also leave plenty of space for you to enforce common sense, and for their actions to be meaningful.

So perhaps regarding the mages tower. You ask them what to do, and they say they'll case the outside. You aren't a big fan of the idea and don't think it's really doable (you sent him away as he is so far above them he'd make their challenges easy, so that makes sense) so for 3&4 you just make up non-descript barriers. High Walls (beyond their skill level), Locked Gates, Heavily Armed Guards, Keep Out signs. If they concur it's reasonable not to attempt it, then you are fine. If they decide to look at getting a massive ladder then you have to deal with that.

5&6, you now have to think of something that makes the castle (possibly) accessible that would be discoverable from looking at the walls (perhaps a 'barely' scalable bit of wall), 1&2 you have to think of an obstacle that would impinge someone walking round the walls might encounter (perhaps a suspicious neighbour).

Then while the balls back in their court, fill in some of the blanks. Hopefully soon, you get back to something there is concrete mechanics for.