While I agree with the other posters, there's another reason this isn't unfair at all:

You're the DM.

You, as the DM, can allow or disallow anything you want, or change any rules you want, for any reason you want. Now, it's bad form to change things on the fly, without warning to your players, if you can possibly avoid it. But even there, it's allowed. And for things like this, where you're deciding in advance, and allowing the players to change their decisions based on the rules you set down? Perfectly fair.