Whisperspell seems to allow you to cast a spell that you don't have to know
How is that? I'm not saying that you're wrong, but i don't understand how to correct it. I thought the phrase "cast a spell" automatically demanded and assumed that this is a spell you know and have access to.

Would a simple change to "when you are casting a harmful, mind affecting spell while hiding and one the same turn that you spend a swift action..." get rid of the problem?