I have written a python 3.4 script (here) to simulate D&D 5e battles.
The aim is determine the victory probability given a lineup of creatures in two or more teams.

The model
Repeating a battle thousands of times gives statistical error of a percentage point and takes a second.
It does not know about space —it assumes everyone is contiguous to everyone, but ignores close combat range disadvantage.
It does not know all of the rules or most of the spells, but it is fairly robust.
The various creatures play munchkinishly by targetting the opponent with the lower AC —it can be switched to random, but I don't think that reflects the game.
The tactics are fixed —a sprawling mess TBH and I would be tempted to make it Bayesian, but it is not really worth it. The tactics are simple: heal/buff if needed, dodge if being targeted and having the edge on turn economy, net if available and turn economy —added for a hypothesis of mine—, otherwise attack. No helping tanks, no grappling barbarians to deactivate their rage or similar.
Some class features are present… that is because I am using my current party as a template.
I can simulate a single battle and get a narration… it sounds pretty much like a normal encounter: a few fumbles, bard gets squashed etc.
It would need a herculean amount of work to make it complete and I am pleased as it is, so I'd probably only do minor tweaks.
The major issue is that casters get pwoned first, which is a big problem. To determine PC merit I did a 'commoner assist test' (assist 3 commoners against 4 giant rats —calibrated for 80% loss if a rubbish adventure comes (hp 8, +2 strength, AC 14)—), which is fairly good.

It has confirmed a few things:
  • A small edge dramatically switches defeat to victory: the contributing factors in order are damage dice, AC and then hp.
  • Everyone knows that the DMG encounter maths are weak. A hard encounter is a 50% win if the players don't use their class extras. If they do it's 90% win. Double the corrected XP of the encounter makes it nearly 50% and that assumes the DM is playing munchkinishly.
  • A 2d6 is the same as a d12+d2–1, d7+d5, 2d8 –2, 3d3+1, 4d2+1 or 7. The extra 0.5 mean advantage of a 2d6 over a 1d12, never gives more than a 4.5% points extra on victory probability when two equal duellist spar (lvl 1–20, AC 14–18). Albeit obvious, this keeps getting debated over the table.
  • 13 commoners is the tipping point to take down a hill giant.
  • Victory probability of a commoner vs. a giant rat is 7% —hence the well known story of the rat killing quest.

Regarding the first and last point: a pretty picture showing how a small change has a big effect.

So, if anyone has any cool hypotheses that they would like tested, please feel free to ask or use the script!