Is spell dmg worth at high levels?

[LudicSavant]

I guess you really just meant that your AC is good enough that 1d6 is a relatively large penalty.

Correct!

Good point about the Manglers vs. Darkness.

Once Wall of Fire is up, what's your plan for inflicting damage, in the "only 2 short rests" scenario? Can't Eldritch Blast now because you can't see the target. Synaptic Static? You may hit the PCs too (depends on positioning but it's only a 20' x 40' area) and penalize your own concentration saves.

Eldritch Blast does not require you to see the target, and since they can't see you, neither side has Advantage or Disadvantage. You can therefore simply pinball them about with Repelling Blast, getting lots of damage off of hazards (like Wall of Fire, Create Bonfire, oil, etc) and repositioning them for ideal AoEs (like Synaptic Static, NOT hitting your allies!).

As an aside, the usefulness of Repelling Blast is also why it's so easy to get multi-hits on Maddening Hex, despite its modest AoE size (though Yam doesn't take that Invocation, it's a resource-efficient damage tool for many Warlocks).