Is spell dmg worth at high levels?

[x3n0n]

playing him basically like any other Barbarian except without Rage [snip]

Lyrandar (Mark of Storms) half-elf Barbarian 1/Celestial Warlock 11. Str 20 (18) Dex 16 (15) Con 17 Int 10 Wis 12 Cha 20 (18). GWM.

I'm sure I'm missing something, but can't tell what it is.

If you're not going to Rage and are going to actually wear armor, why Barb 1 (especially vs your originally-planned Fighter 1)?