It all depends on what you are.
Magic helps because they can use miss chances to start negating the ability to actually hit you. Stuff like Mirror Image, Blur/displacement, Blink, etc.
Most chargers don't have great saves and, as such, can be hit with debilitating spells (bestow curse, slow, blind effects, solid fog, etc.).
The feat Elusive Target would be another great choice. It's one of those niche feats that allows the user to negate power attack from the attacker. This doesn't completely stop the attacker, but greatly reduces his damage.
There are also feats and class features that allow you to side step charges.
A combination of these things would be best.
Edit: Almost forgot line-of-sight eliminators. Walls are wonderful. Even a Warlock with Wall of Gloom can put a real hinderence on a charger (anyone that needs line of sight, really). Even Silent Image walls can help with this issue.