Yeah, that's completely understandable and despite the fact that I might consider making an exception I do agree. In the event that you're making a check that is important enough to warrant this type of diligence I would expect a better buff to be available that doesn't require a ruling like this.
This is a purely mechanical viewing of the rules though, I find the narrative reasoning that PhoenixPhyre offers to also be persuasive.