PDA

View Full Version : Game Theory: Quantitatively Measuring Systems by Traits



Tyndmyr
2011-04-29, 03:01 PM
Ever argue over if a system is combat heavy, PvP heavy, or otherwise focused on another specific attribute of gameplay more/less than other games? No more. While I can't answer if a game is good or not, I can answer how much a game focuses on a given aspect of gameplay.

Step 1. Determine exactly what trait you wish to test for, and which systems you want to test. For the initial example, we will use Combat as our focus, and the 3.5 and 4e phb's as our test set.

Step 2. Determine how you will judge positives and negatives, and what to do about things that are both.

In this case, any power that explicitly has a non-combat use described for it in the text referenced will be considered something that has non-combat utility. For instance, the 3.5 spell Mending is considered a clearly non-combat power.

Any rule that makes no such reference, and is described only in a combat context is a combat power. An example is "Disrupt Undead" in 3.5, which includes no reference to out of combat uses.

An ability or power that has both will be, for this test, counted as a non-combat power. This is for simplicities sake as while purely combat spells are rarely useful out of combat, non-combat spells can frequently be utilized in combat. Any given trait will probably have an area of overlap such as this, and it needs to be defined such that it will be adjudicated equally for both. For example, 3.5's Fireball has both clear combat rules, but also has enumerated non-combat uses such as melting metals.

Additionally, consider how you will rate choices of choices. For instance, a feat can be rated easily as a combat feat or a non-combat feat, but how do you rate a class ability that gives you a free feat? In this case, we are not rating them because it's unfair to count the same thing repeatedly, and we will already be rating feats and spells on an individual basis.

Lastly, consider how to mark things that are not clearly either. Typically, as in this example, they will be ignored. An example from 3.5 would be the Size Category(Medium) listed in the human racial bonus. While you could possibly make an argument for size categories being relevant to either combat or non-combat, it's primary purpose is as an attribute of the character, much like skin color would be, and it is not a separate choice or ability. It therefore cannot be counted as a separate option.

As a side note, remember, we are simply enumerating choices, we are not rating choices. We are uninterested in the comparative, highly subjective, values. If a +4 to diplomacy is better than a +2 hit points is not the goal of this. We are simply interested in the quantity of a given trait in a system.

Step 3. Determine your sources. It is highly encouraged that you choose equivalent books when at all possible. Comparing a monster manual from one edition to a phb from another will yield garbage results. Comparing one phb against another is a vastly better test. Note that nearly all systems have something approaching a phb and/or a dungeon masters guide, though some, like pathfinder, combine them into one book. It would thus be fair to compare the 3.5 PHB and DMG to pathfinder's core rulebook. Note that in some systems, like CoC, monsters are included in the core rulebook. In such cases, you must either eliminate that portion of the book from your comparison or include the comparable portion of the system you are comparing against.

Step 4. Compare Rules to Rules. For all rules based comparisons, you will ignore all solely fluff based areas such as the common sections entitled "What is roleplaying?" and the like, because they are not relevant. This does not exclude areas that mingle fluff and rules, like power descriptions. If you're actually attempting to measure the ratio of fluff to crunch, obviously you'll not wish to exclude anything.

Step 5. Normalize the results. Obviously, some systems are denser than others. The pathfinder bestiary book is much larger physically than the 3.5 monster manual. If I want to find which system is more dragon-focused, it would not be fair to simply tally the number of dragons in each, but instead, to tally the number of dragons and non-dragons and find the percentage of the total for each. Whichever one has a higher percentage of dragons in it is a more dragon-focused book.

Step 6. Crunch the numbers. Aright, this part is a bit rough. Gotta do a lotta flipping through and counting. Present results as broken down as possible, with a nice, simple percentage summary of the system at the end. This allows comparison with as little grief and bias as possible.

This enables you to rate a game as say:

Dragonrama
37% crunch
15% pvp
80% combat
90% dragons

So long as you utilize the same method when rating them all, you can objectively measure the relative focuses of games, which makes them easily to quickly describe to your friends, and allows you to investigate other games which interest you based on their similarity to or differences from games you already play.

Tyndmyr
2011-04-29, 03:02 PM
**Reserved for Detailed 3.5 Eval**

We will not be counting the default class skills, etc yet. I might put skills into their own section, and it'd be silly to count them twice.

Classes:
Barbarian - combat/2 non combat -Seriously, not surprised.
Fast Movement: Combat
Illiteracy: Noncom
Rage: Combat
Trap Sense: Combat -Traps are primarily combative in their function. They form encounters, and thus, any time you run into one it is indistinguishable from combat, even if an only brief one.
Uncanny Dodge: Combat
Improved Uncanny Dodge: Combat
Damage Reduction: Combat
Greater Rage: Combat
Indomitable Will: Combat
Tireless Rage: Non-Combat(because post-rage is often a non-combat time)
Mighty Rage: Combat

Bard - 1 combat, 2 noncombat

Spells - Not counted. Spells are choices in their own right.
Bardic Knowledge - Non combat
Countersong - combat
Bardic Music - Non combat (only being listed as one ability, since it is in the text, and text trumps table. If you were rating versatile powers seperately, this would absolutely count).

Cleric- 2 Combat, 3 Noncombat

Aura - Noncombat
Spellcasting - Not counted. I'm gonna skip listing it from now on for brevity.
Diety, Domain & Domain Spells - Noncombat(due to domain ability/diety. The spells are not relevant due to being separate)
Spont Casting - Combat(it's purely cures and inflicts. While healing can exist out of combat, all directly hp affecting spells with no other affect are inherently combat-related)
Chaotic, Evil, Good, Lawful Spells - Noncombat.
Turn or Rebuke Undead - Combat


Druid- 11 Noncombat, 1 combat

Spont Casting - Noncombat
Chaotic, Evil, Good, Lawful Spells - Noncombat.
Animal Companion - Noncombat.
Nature Sense - Noncombat.
Wild Empathy - Noncombat.
Woodland Strike - Noncombat
Trackless Step - Noncombat.
Resist Nature's Lure - Combat(bonus to all saving throws would be a non-combat ability, but bonus to spells targetted at you? Yeah, that sounds like combat).
Wild Shape - Noncombat(again, if you're looking for versatile...)
Venom Immunity - Noncombat (Due to inhaled/ingested poisons)
A Thousand Faces - Noncombat
Timeless Body - Noncombat

Tyndmyr
2011-04-29, 03:06 PM
**Reserved for Detailed 4e Eval**

valadil
2011-04-29, 03:11 PM
There was a discussion on measuring 4e complexity (http://rpg.stackexchange.com/q/6111/819) on stackexchange a while ago. You might find it relevant.

I'd consider counting powers with combat and non-combat applications separately as versatile powers. Wall of Stone is a weaker wall than Force, but I always take it for its utility abilities.

Tyndmyr
2011-04-29, 03:29 PM
That'll certainly make interesting reading, at any rate.

So far, I'm getting a remarkably solid link between non-combat options available and tier. I suppose I should have expected that. Tallying these all is likely going to take a while.

Edit: I thought about it...it's interesting, as it gives more detailed information about versatility, but it would screw with the goal of having a list of percentages for each trait. Hmmm, perhaps I could put "versatile powers" as a trait for RPGs. Might be useful, I dunno.

valadil
2011-04-29, 03:35 PM
Tallying these all is likely going to take a while.

How are you evaluating if a power is combat worthy? With a local copy of the DDI compendium content I think you could script away some of the tallying. Any power that includes "attack" or "damage" goes in the combat bucket and you don't have to look at it anymore. Then just read through the remaining powers.

For instance I get 2805 powers that include "attack" out of 5097 power files. That cuts out a lot of reading on your part. I'd expect there to be other keywords that can quickly mark something as a combat power. Not sure if any terms could indicate a utility, or if those would have to be weeded out of whatever is left over.

evirus
2011-04-29, 03:41 PM
How are you evaluating if a power is combat worthy? With a local copy of the DDI compendium content I think you could script away some of the tallying. Any power that includes "attack" or "damage" goes in the combat bucket and you don't have to look at it anymore. Then just read through the remaining powers.

For instance I get 2805 powers that include "attack" out of 5097 power files. That cuts out a lot of reading on your part. I'd expect there to be other keywords that can quickly mark something as a combat power. Not sure if any terms could indicate a utility, or if those would have to be weeded out of whatever is left over.

I think that will also grab feats that also give bonus to attacks. Say the Weapon feats that should be listed as "versatile" since they give skill boosts as well.

EDIT: OOPS! You're only looking up powers. Sorry. Are there any powers that have both attack and non attack functions?

valadil
2011-04-29, 03:45 PM
EDIT: OOPS! You're only looking up powers. Sorry. Are there any powers that have both attack and non attack functions?

Off the top of my head, yes. My group's Avenger has a power that gives her a fly speed for two rounds. It has solved more physical obstacles than combat situations. Although if there were any powers that gave an attack and a fly speed, my metric of just grepping for the attack would fail to take into account the non-combat use.

Tyndmyr
2011-04-29, 03:47 PM
I think that will also grab feats that also give bonus to attacks. Say the Weapon feats that should be listed as "versatile" since they give skill boosts as well.

EDIT: OOPS! You're only looking up powers. Sorry. Are there any powers that have both attack and non attack functions?

Oh, I'll be going through feats and class abilities and rituals and such as well. I honestly don't remember all of those off the top of my head, Ima have to go through the book, but I know there's a coupla things that could be used for both. Probably less than 3.5, though, 4e moved away from open ended abilities in a more defined direction.

Indon
2011-04-29, 03:51 PM
For measuring systems for comparison purposes, I wonder about a scenario:mechanic model, where a set of generic scenarios are modeled ("A player character wants to kill or otherwise defeat a different player character," for example) and the mechanics used to reconcile with the scenario are analyzed comparatively.

valadil
2011-04-29, 03:55 PM
For measuring systems for comparison purposes, I wonder about a scenario:mechanic model, where a set of generic scenarios are modeled ("A player character wants to kill or otherwise defeat a different player character," for example) and the mechanics used to reconcile with the scenario are analyzed comparatively.

I don't think it's possible to reach consensus on how common various scenarios are.

Urpriest
2011-04-29, 05:09 PM
You've given some thought into how to discretize powers (excluding spells and bonus feat choices from class features), but ultimately your system is still far too subjective. Even elements with the same name in similar systems, like Feats in 3.5 and 4e, have radically different roles in gameplay that are independent of the number of them in the system (your normalization condition). Let alone the difference between attack powers and spells.

Instead of normalizing based on the number of game elements, you might improve your comparison by normalizing on how frequently those game elements occur in an n-hour gaming session. Of course, then you would face a significant amount of dependence on the group involved, so you'd want to correct for that in some way.

Bang!
2011-04-29, 05:27 PM
Ever argue over if a system is combat heavy, PvP heavy, or otherwise focused on another specific attribute of gameplay more/less than other games? No more. While I can't answer if a game is good or not, I can answer how much a game focuses on a given aspect of gameplay.
While I like the idea and am interested in how these add up, you're comparing verbiage rather than actual gameplay emphasis. This might roughly work with games that are about the same things (like different editions of D&D), but I believe this is a flawed measure for systems with dissimilar priorities and structures.

Hypothetical:
A game that has rules for western shootouts but that summarizes those scenes with a single coin toss isn't a game about western shootouts. It's a game that acknowledges western shootouts are going to happen, and that provides a rudimentary mechanic to keep the game moving when they do, but the emphasis of the game is going to gravitate toward areas with a greater rewards and a more engaging metagame.

If it were a class system, it could even have a few classes with abilities that incrementally modify the probabilities of the coin toss. The resolution mechanic would remain just as dry, the most interesting elements of the metagame would not be altered.

Side Note:
The title made me think this thread was going to be about something very different. :/

NichG
2011-04-29, 07:56 PM
This is table-dependent of course, but what about having a series of stopwatches and measure how much screen time is spent on various aspects of the game over the course of a year-long campaign or something?

If you then look at the extrema of that result across several tables, you get both the mean focus of the game and also how far it can be pushed.

So D&D might end up being 'Combat: 20%-95%, mean 60%, PC-NPC RP: 5-50%, mean 35%', problem solving: ...

erikun
2011-04-29, 10:37 PM
While I think you have a good idea here, I would personally not be interested in partitioning everything into simplified categories. As you pointed out, some abilities are useful in and out of combat. Fly is good for avoiding melee, and for getting up cliffs. A Shadowdancer's Shadow jump ability is useful both for getting around and for hit-and-run in combat. I've personally used Wild Shape for hunting and remaining unobtrusive at least as often as going into combat.

Overall, I am less interested in statistics like "70% of the spell list is for combat, 30% utility" and far more interested in "75% of spells can be used in combat, 25% can be used in social situations, and 65% can be used in general out-of-combat situations." This information is far more valuable to me, and rather than ranking how 'combat-y' a particular system is.

Indon
2011-04-30, 09:46 AM
I don't think it's possible to reach consensus on how common various scenarios are.

You shouldn't need to. Players should be able to look at a set of scenarios and go, "These are the ones in my game, system X, Y, and/or Z handles this set the best, perhaps I should try that."

Aux-Ash
2011-04-30, 11:16 AM
I'd like to add my voice to those that questions this approach to classifying systems. While certainly, it seems to be able to differentiate certain "powers" (or whatever you wish to call them) into two groups. Does that tell us anything useful?

First I question the wisdom of using wether certain powers are combat or non-combat as a useful metric to determine how combat-intensive a game would be. If system A has 80% of the powers as non-combat but 90 % of the time is devoted to the remaining 10 %. Is it less combat intensive than system B with 90 % combat powers where 40 % of the time is spent on non-combat challenges?
And what of games with very few powers or that is entirely powerless?
If you want to compare how much a game focuses on certain aspects, shouldn't it look at what is in the toolbox for that aspect as opposed to what each tool does? How available these resources are? The complexity and support provided for different types challenges? And so on...

Moreover, you go for a strict non-combat/combat division between the powers (in my meaning the fact that you could do that tells us more about what the system focuses on than the exact figures does). Yet you ackowledge that there is an overlap yet lump those as non-combat for simplicity. I'd argue this cause a misrepresentation, an oversimplification. Suggesting the system is more binary than it is. Since these powers clearly can do both, shouldn't they have a category of their own? Even if they are few indeed.

There's also no accounting for the number of powers provided. If system A has 20 % of 5000 powers it has a lot more support than System B's 60 % of 200. If you truly want to show us a comparison a normalisation of the numbers would have to be done (ex. 60 out of every 100 powers).

Moreover, you say you ignore the fluff-based sections because they are not useful. I absolutely disagree. It is very important in matters beyond comparison of fluff vs. crunch for certain systems. Notably systems deeply atmospheric, focused on a single world and so on. Where the fluff might be an integral part in certain aspects of the rules.

Finally. I don't think percentages is a useful presentation of what you intend to show us. It is a crude tool only useful for the description of comparative sizes. You might want to look into more advanced statistics for a proper numerical presentation.

I don't mean to just complain and critiscise your work. I'd love to see something actually useful be developed.

valadil
2011-04-30, 11:34 AM
...but 90 % of the time is devoted to the remaining 10 %. Is it less combat intensive than system B with 90 % combat powers where 40 % of the time is spent on non-combat challenges?

How much time gets devoted to combat or non-combat is going to vary with the group and GM. You can't measure it without all the other GM's saying "that's not how my games run!" It's subjective. You can measure how many combat/non-combat/versatile abilities exist in the game though. I think you can also measure if those abilities are mutually exclusive (say, you have two slots, put any power in them) or if you don't have to choose between the two (ie, you have a combat slot and a non-combat slot, pick a power for each). Not sure if that relates to this measurement though.


You shouldn't need to. Players should be able to look at a set of scenarios and go, "These are the ones in my game, system X, Y, and/or Z handles this set the best, perhaps I should try that."

Ah, that makes sense. I misunderstood what you'd be using the stats for.

Aux-Ash
2011-04-30, 01:28 PM
How much time gets devoted to combat or non-combat is going to vary with the group and GM. You can't measure it without all the other GM's saying "that's not how my games run!" It's subjective. You can measure how many combat/non-combat/versatile abilities exist in the game though. I think you can also measure if those abilities are mutually exclusive (say, you have two slots, put any power in them) or if you don't have to choose between the two (ie, you have a combat slot and a non-combat slot, pick a power for each). Not sure if that relates to this measurement though.

Yes, how much time is devoted to various tasks will vary between groups. But how long solving the tasks with the tools available will take is a much less variable figure. Most games have very extensive support for combat, allowing complex and variable ways to fight. Few games provide the same framework for social, non-combat-physical, stealth or production related challenges though.

If non-combat challenges often relies on a single die-roll when combat relies on choosing the right tactics, preparation and some luck, there's no question which of the two will take up the greater amount of time (assuming equally many challenges).

So rather than measure how many individual powers that may or may not support one thing, measure the underlying complexity and support of the mechanics for challenge-resolution. Divided in more groups than just combat and non-combat.

Indon
2011-04-30, 10:12 PM
So, I'd propose a woefully incomplete list of metrics by my scenario proposal:

Each scenario should be, ideally, analyzed for the following factors:
-Does the game have mechanics for this at all, or is it expected to be handwaved if it comes up and/or is irrelevant due to the system's assumptions?
-Are the mechanics for this expedient - if you're resolving for speed, just trying to move things along, is it faster to invoke the mechanics, or to introduce some quick GM system hack to expedite things?
-Are the mechanics for this dynamic - if your players want to solve a problem under this mechanic in a meaningfully creative way, is it beneficial to the players to do it?
-Are the mechanics for this thorough - can they take into account details that should have an impact on the resolution of the mechanic?

So for instance, combat in 4E when it came out was somewhat dynamic because the system outlines 'tricks' players can do in combat, was thorough because the rules cover a lot of things, and was not expedient because combats tended to keep going even after the mobs stopped being threats and it was better for the DM to handwave the rest of the combat.

We could, additionally, stand to analyze a set of fairly common 'toolset' attributes, like character creation time.

Scenarios:
-A player/NPC fight.
-A player/player fight, if it does not use the same mechanics.
-A noncombat player/NPC contest (like drinking or arm wrestling).
-A noncombat player/player contest, if it does not use the same mechanics.
-The players travel a long distance.
-The players explore an area, dealing with its' natural hazards.
-The players try to save someone at the verge of death.

Toolset Factors:
-Cost of enough books to make the game reasonably playable.
-Average character creation time.
-Randomness factor in resolution mechanics.

Other proposed scenarios/factors would be appreciated.

Tyndmyr
2011-05-02, 11:26 AM
You've given some thought into how to discretize powers (excluding spells and bonus feat choices from class features), but ultimately your system is still far too subjective. Even elements with the same name in similar systems, like Feats in 3.5 and 4e, have radically different roles in gameplay that are independent of the number of them in the system (your normalization condition). Let alone the difference between attack powers and spells.

Instead of normalizing based on the number of game elements, you might improve your comparison by normalizing on how frequently those game elements occur in an n-hour gaming session. Of course, then you would face a significant amount of dependence on the group involved, so you'd want to correct for that in some way.

Almost all RPG systems include enough choice in mechanics that playstyle will drastically impact the ratio at which the mechanics get used. Consider 3.5...I've played in both all melee and all caster groups. Different mechanics got very, very different amounts of playtime in them.

Basically, there is no standard group. Any method of assigning a "normal" group to each of them will inherently be biased, and you'll get final results that are not terribly applicable to anyone but that group.

Hence the treating of all options as equal. My goal is not to judge the wisdom or relative value of the options, but merely to measure the options that exist. This will not measure, say, the level of balance in 3.5. It's just not useful for that.

I want to avoid any table-dependence at all. I feel like as soon as you include that, the entire thing becomes an exercise in "well, I don't play like THAT". Taking 3.5 as an example, the level of combat in a campaign can vary wildly.

Adding social/non social in addition to combat, non combat is also good. I prefer to list them separately because there is a great deal of overlap. So, a flat percentage for each is probably best.