Fair enough.

What is the party composition? I like the idea of negative XP. What if they lose XP everytime they cast a spell? The higher level the spell, the more XP they lose?

Obviously, this will hurt full casters more than anyone, and pure melee classes will be unaffected. An alternative might be needed for them, if you (or anyone else) can think of it.