So, another option:

Say yes, and have them do legwork to figure out what's going into the tower... and have them help you build it.

Obviously, he's going to have defenses... if the players say he's not, they're being silly.* What might those defenses look like? Have them give a few options. What might his servitors look like? Get the players to build his defenses for you in the course of the game, decide which of their options to take, and then present them with the results at the next game.

*A fun option, from Mary Kirchoff's "Night of the Eye", is that the mage HAS no defenses on his villa. It's just a place he stays, and occasionally has people over. He's actually got a teleportation circle in there, though, that takes him to his REAL home, where all of his valuables are kept.