Well, you want to play to level 20, so, let's see what that does.

Let's assume a fairly unoptimized barbarian, on a charge. He uses a greataxe (1d12) for flavour, has 24 strength (not many magical items), power attacks for full and rages for another +6 strength. Somewhere, he also got the ability to get x3 damage for his power attack.

Damage: 1d12+13+60, or about 80 damage on average. That's 8 ability damage. Fair enough.

To compare, a level 4 (new level 1) rogue: 1d6 (shortspear for constitution)+2d6 (sneak attack). He's two-weapon fighting, so gets two attacks. That gives him 6d6 damage, or about 21 on average. 2 ability damage. The 2 constitution damage makes this about 4 more damage.

If he fights a fighter with 18 constitution, the damage will kill that guy long before the con-loss does.

So, doesn't seem to bad. I'm not sure what happens if you include spells for ability damage, though. Shivering touch is bad enough on it's own...