Even with just leather or hide, you are no worse than most warlocks or non-dragon sorcerers