3.x Alternative Combat Rules

[Waargh!]

Here are some suggestions on making classes function more on their role. You can add more on this thread

Attack of Opportunity
When succeeding (deal damage) with an AoO the action that provoked the AoO needs a check to be executed, similar to the concentration check for spells. For skill checks disrupted, use that skill check to try to avoid losing teh action. If it involves an attack roll (ranged attack) then use an attack roll with the same bonuses to determine success.
The DC is equal to the damage dealt from the AoO.
Creatures get an AoO per combatant. With Combat Reflexes they gain +DEX AoO per combatant.
Note: actions that don't require a check, like moving away, are executed normally.

Removing anti-AoO options: Casting defensively and feat/class abilities that enable you to avoid an AoO will not be available. Tumble will not be available either to avoid AoO. If the are a class future, they will have to be replaced with a bonus feat. Feats/class abilities that let you take 10, perform an action without a check are still usable. If a normal action, like casting a quickened spell, is specified that it doesn't provoke an AoO, it doesn't provoke an AoO.

Result: Melee characters will have more control over ranged characters or spell casters. The later will have to move away and then attack, giving the melee character an advantage. So ranged characters will have to be careful on where they are positioned. Anti-AoO options should not be available if you choose this path. If you like the rules behind AoO, why break the rules?

Following up / Moving away
You can ready your move action. You can choose to trigger you move action after a move action of a target or as when a target moves at a specific distance or square. If the creature uses teleportation, takes a 5-foot step or tumbles, your move action is not triggered. You can choose to trigger it after this move.

Result: you can choose to follow up a character so you eliminate the "freeze" effect (some characters moving, while the others standing). This can be used for melee characters to lock targets, or for ranged characters to be able to move away when enemy approaches (skirmish).

[Glimbur]

This proposed fix doesn't do anything about 5' steps. Or various swift action teleports, but (almost) everyone can use a 5' step and it will get you away from an AoO in a majority of situations.
Ways to stop the 5' step:
1) Have reach and have the enemy inside your reach, not at the edge. Since very few weapons allow attacking both at reach and not-reach, this means you should be at least Large... or have natural reach of at least a Large thing.

2)Positioning. If there's a handy wall preventing the spellcaster/archer from moving backwards, then they can't.

3)Difficult terrain. Knights get a useful ability, and I think there's a ToB stance that also helps. Battlefield prep with marbles or whatever is action inefficient... you could instead spend that time killing the opponent directly.
3a)Fight underwater or on the side of a mountain or somewhere else where you have a listed move speed but your foe doesn't. Not always viable.

Following up, as you suggest it, will only help versus... nothing, because if you're moving away with more than a 5' step you will full withdraw (http://www.d20srd.org/srd/combat/actionsInCombat.htm#withdraw) to avoid the AoO, and a double move is generally further than a single move. The end result is the melee type is some distance, generally 10' to 30', away from the person who wants to retreat. Sure, you can charge, but then you can't ready a move action. It's somewhat helpful, but doesn't actually solve the stated problem.

[Waargh!]

You can simply flank the target not enabling him to escape with a 5-ft movement. You can use a follow up action against casters/archers that take a 5-ft step back. Considering you can deal a good amount of damage with an attack, they can lose their action. Even if you don't follow up, you can at least full-attack a caster. That fixes the problem of a two-weapon fighter trying to use his full-capabilities against a caster. Even if the caster doesn't tumble, you will attack twice per round (AoO + standard action), rather than full-attack. The combination of the both make it viable to stop casters/archers.

The penalty for chasing around is the move-action. So allied positions will play a vital role, preventing others to chase around on the artillery units.
In a 1-vs-1, with Combat Reflexes, you can typically gain +2 attacks on someone moving away from you if he has a greater speed than you. If he has lower, then you can always threaten him. If you dmg is greater than his attack, you can simply stop him from acting. The check assures that high-level characters can get away with it from low-level creatures. But a high-level monster that ambushes a party from the rear will be a problem to the archers/casters.

Situations fixed
1) Melee vs Ranged. The ranged character moves back and shoots. Considering that they both move, this is highly difficult to simulate without the following up action
2) Trying to fire an arrow in the face of a creatures swinging his sword at you. If he manages to strike you there is a high chance of you losing your action.
3) Breaking the rules. Tumbling has too low a DC. At 1st even if Tumble is cross-class and you have +0 DEX, you will get a 40% chance of tumbling (4 SP). If it is and you get a +3 DEX, you will have a 70% chance. Both are too high. Defensive casting simply breaks the rules as well. The DC is mid-high level is too low. Class futures/feats that make the archer function in melee combat kind of change its role, which I don't find a good thing.

[Glimbur]

Following up / Moving away
...If the creature uses teleportation, takes a 5-foot step or tumbles, your move action is not triggered.

As you have currently written the rules, the 5' step is still very powerful.

I don't see why you would get any attacks on someone moving away from you unless you have reach and they are not at the edge but inside. They could just take a withdraw (http://www.d20srd.org/srd/combat/actionsInCombat.htm#withdraw) action and be charge distance away. However, a withdraw does not have to be in a straight line but a charge does. If they simply move around a corner or with a tree between you and them, then they will get away barring you being faster than them. If you want to change the withdraw action, include that in your rule set. Making it a full round action but only allowing you to move your speed instead of twice your speed would significantly reduce its power... not that it's very powerful to begin with because any action not spent directly on killing your foes is... questionable.

Flanking is very helpful. These proposed changes make it even more helpful.