A bunch of people are missing that being in swarm form costs you all Special Attacks of the base form.
That means no spells, no psionics, no Eldritch Blasts, maybe no incarnum (but you've probably lost all your chakras except Totem and Soul anyway1)... on top of the "no normal attacks" rule you already had.
Technically, this wouldn't apply to class features you got after acquiring the template, though.
1. Unless you're a body part swarm, I guess. Which is kind of hilarious.