It doesn't advance your wizard (or, for that matter, any other) spell levels at all. That's a problem that at least EK doesn't have. A relatively high level gish with only 4th level spells (from a limited selection) isn't going to be nearly as effective as almost any other build.
It also clashes with established lore. Josidiah Starym, probably the most famous bladesinger from the Realms, became Spell-Major of Myth Drannor — essentially the most powerful martial magic-user in a really heavy magic area. He didn't do it with 4th level spells.