Last time I looked XP was your aquired experience. A peasant has as much of life energy as a level 20 wizard. I agree it's tiring to do that stuff that's why I did new penalties.
If a peasant had as much life energy as a godlike wizard, he would be just as easy to kill. He isn't. Also, your XP represents the strength of your soul/personality/self/insert-your-term-here, which naturally gets stronger by overcoming great challenges (generally by adventuring). When you use powerful magic, some of yourself goes into it, so you lose XP.