I've mulled over whether or not Shadow Familiar should be an automatic part of the class, I'm still not sure. At the very least, there are no dead levels in a caster class, which always amasses more of what it casts. That's why there'll be blank levels, and why no one should feel obligated to completely cram non-specialized casting classes with class features.
But for the fluff idea you have for the familiar, it's more or less already captured in the Master of Shadow prestige class, and is an easy refluff in other forms.