As far as I can tell the reason to have Spell Tomes is so enemies can drop them. I'd be okay with keeping the tomes if they only dropped from enemies/loot chests.