Okay, you convinced me, but I am doing a slight twist on it:

-The Bullrush/Trip/Disarm bonus comes in at level 5 as you suggested, with a bonus just to int mod. Rather than completely negating enemy size modifiers, instead he reduces enemy's effective size modifier by 1, to a minimum of Medium. At level 9 this reduces the modifier by 2, to a minimum of small. Given the exponential scaling of those modifiers, this helps level the playing field a lot, but not quite to the "not effective at all" level.

-Penetrating Strike changed as discussed, grants an increased crit multiplier, and increases precision dice damage on a crit. Still considering adding a bonus to precise strike to make it a bit more potent, how much it's needed depends largely on optimization level. Just going with pure duelist, it's probably needed, but if you're picking up say that dex to damage feat from ToB, or Int to damage from swashbuckler, or both, it's probably not really necessary. So I guess the question there is if I should rely on outside influences that aren't guaranteed to help the class keep up, or should the class be more independent, and simply that much stronger with the synergy from the other sources.

-Changed Improved Reaction so it requires no armor at all. Then changed Canny Defense and Elaborate Parry to work with light armor (but restricted to 1 per level for canny defense and 1/3 level round up for Elaborate Parry). This should reinforce being armorless as the default option, but allow light armor for those who want cheaper access to armor/armor upgrades. Light Armor may be better when you first enter the class, but armorless should pull well ahead as you get deeper into it, and better at what you do.

I think that's all the changes. For right now, I'm still inclined to keep the dimensional anchor effect. If someone wants to teleport away, let them use the extra actions to move away first. If more people feel it really doesn't fit the flavor of the class at all, I'll remove it, but personally I think my justification that they are able to interrupt a target within reach from teleporting is fine.