View Full Version : How to beta-test.

2012-05-22, 11:41 AM
Hello again everyone. This time the question is quite simple. I've reached a point in designing my "indie" game where I need to expand for some beta testing. Problem is that the game is meant to be played in a very specific way, tied in to the setting, for which I still haven't had time to write all the material I've thought of, thus making it necessary that I personally participate in all games. Though it is manageable, the question is what are the perils and pitfalls of such an endeavour. For one, the game is specifically designed for an IM environment, but I wouldn't like copies of the rules equal to the number of testers floating around with people I'm not acquainted.

2012-05-22, 03:59 PM
This is a pretty vague question to answer, because without even the idea of how the game is intended to be played, it can be hard to determine how detrimental playing the game without the rules would be.

I've never participated in a beta test, but I would assume that you should be testing with some specific questions in mind. That is, have some idea of what you want to ask the players after the session. If the goal is to ensure that combat runs smoothly but know that weapon variety is sparce, then make sure to ask about combat pacing but not a lot of questions about the variety of weapons.

I'd recommend keeping your questions open-ended and accepting all forms or feedback. If you know your population statistics are out of wack, then don't respond with "I know the populations statistics don't work yet, but..." every time someone brings it up. Just note it and move on.

Ideally, I think that you should have pre-generated characters for players during the first playtest. Depending on how the game system works, you could either include game-relevant explanations for each section, or just interpret it all through your GM screen.

Future beta tests would probably include character-building information, to see how easy and comprehensive those rules are. Preferably it would be with a different group, who aren't familiar with how a properly created character "should" look, so you get some idea of what someone looking at the ruleset would put together.

You aren't going to be able to avoid giving away the rules, at least not significant sections of the rules, during playtesting. I'd suggest testing with people you can trust, or at least including other people into the design team to bounce ideas off of.

2012-05-22, 04:52 PM
also keep in mind when testing that abilities that are to vague as too their use need extra attention to make sure that they aren't underpowered or overpowered, an example of where this was forgotten would be things such as shapechange, wildshape and polymorph in dnd. Also remember to take in to account at least some non-standard style characters(for the setting anyway, for example if the base game has mostly blasty mages, then try out buffing, summoning, or debuff mages, or if they base game has mainly buffing, summoning or debuff mages, trying out a blasty mage could be useful to help test the system's balance), at least after you finish the first part of beta testing.

2012-05-22, 08:47 PM
So I'm a beta tester for a few products (and doing more as we speak) along with studying enough beta testing on the software development side of things (which doesn't exactly cross over but some principles still apply) so I feel I can drop a few comments in here.

Firstly you need to ask yourself why your beta testing, not simply to improve the quality of the game but more specific. Are you looking for number crunching to make sure your math is right? Are you trying to test the clarity of your written work? The smoothness of the combat? The skill system? Do the mechanics match the theme? etc

Depending on what question you want answered (ultimately its best to test everything but one step of a time) is what and how you need your beta testing to go. The big thing is that your going to have to give other people copies of the game in order to test it. You dont need to give everything to everyone but it damages how your testers will treat the game versus the real world. Beta testing is ultimately designed to reflect reality and example the consequences of this environment on your product. The biggest one is that people are learning and applying your system with nothing but the text to teach them how and why.

Now you can try the 'GM gets the rules' approach giving a copy of the game to them and giving premade characters or something to the players and the GM gives the rules are the players hit a situation where they need it. This is a GREAT way to identify what skills tend to get prioritised and how your classes or skill systems work to direct players, its even a good opportunity to explore the game world (if you provide a premade scenario too). You do get some testing on the written text and how your design goals have transfer to an isolated entity but it only lets 1 person from the whole group actually look at this stuff. Given its a GM they have tended to spend a lot of their time deciphering poorly written RPGs so your getting biased data while players generally come from the side where some explanation is provided but dont do as much deciphering as a GM (on average). This can result in a game which is very poorly written towards a player and creating systems where certain skills are passed off because you've directly players into certain direction simply because of your scenario.

While you I won't say you can't beta test without giving out most of the rules along with atleast a first draft of the fluff (though preferrably more polished than that) your really doing as much harm as help. As a tester the worst stuff I ever see is the dumping of rules and fluff. I'm not the math cruncher in the group and fluff without context is often not particularly interesting. When a pre-made scenario comes in everything becomes much more valid, the fluff makes sense and is applicable and the rules are not just a bunch of numbers in a book but a mechanism to let me roleplay and as a result I get to actually experiment without confined conditions (which is what you want out of a beta test).

Im not trying to be rude I completely understand your concerns but a beta test needs to come from a mostly complete base that can contain many errors and mistakes but ultimately can operate independent of the developers concerns, particularly if you want a specific playstyle to be used (which is all about how your rules work AND how you write them)


Until your ready for independent people to get access to your product (both from a completeness sense and having your copy floating around sense) your not ready for a beta test.

2012-05-22, 10:46 PM
Another thing to be careful of is that many aspects of the game will run more smoothly than they would without you there because you will be present in all of the games. You should try to make a note of everything you find yourself clarifying and check later if it's explained in the rules and they just didn't read that part, or if you need to put more clarity in the written rules.

I've been doing some testing, recently, mostly with my in-laws whom I trust enough to give the rules freely, but I can tell you this, most of them didn't read more than a small part. It might be hard to find a lot of trustworthy play testers willing to read through a load of rules.

2012-05-23, 08:28 AM
In the hopes of clarifying a few questions, I will add some information about the game and setting. The game is supposed to be rules-light (20-30 pages) and is divided in two variants. The first is a diceless mixture of elements from oWoD, D&D and others, while the second is a gmless storytelling game where collaborative narration is forwarded by use of ritual phrases and interpretation of tarot cards. The main problem I have seen so far is people having trouble adjusting to the DIY general philosophy of the rules, which would be improvising, adding and removing whatever they see fit.

The other issue is the setting. Stories are mostly focused in a detailed continent over the course of ~8000 years. I have intentionally left the world and timeline open-ended enough for anyone to make additions. Still there is again a design issue that people have some difficulty adjusting to. The setting functions with real world natural laws, or rough approximations thereof, with one exception: Analysis, which would be a sort of "magic" but in effect closer to large scale chemistry or quantum mechanical manipulations. This would mean that all species can/should be explained by evolution, technological advancement plays a crucial role, and all in all players are required to have at the very least some fundamental understanding of science to work with Analysis, more so the GM who would be advised to have at least a university level knowledge.

[will add some comments on material distribution later]

2012-05-23, 09:52 AM
Additionally to what Rorrik said, you should try to test the game with players from different backgrounds (maybe even people that never played RPGs). If all your testing players are also D&D players, some mechanics or concepts that are unclear in your rules but evident to a veteran player may go unnoticed.

2012-06-04, 08:02 AM
*Casts: Resurrect thread*
So, fellows, I needed to add two more questions:
1) Is there any benefit/protection for game material in Creative Commons? Even though I'd hate to use any sort of copyright, I'd like the material to stay confidential until it's released.
2) How can anyone find beta testers, apart from the looking for group forums?

2012-06-04, 08:12 AM
I don't have much advice to give, but perhaps this video (http://penny-arcade.com/patv/episode/playtesting)can be of some help :smallsmile:

2012-06-04, 08:28 AM
I've been putting some thought into the testing part of this. I don't plan to make money off my game though, so the distribution side of things is another matter.

You should explicitly test a game that you run and a game that you don't run. You need to see how the game works when run as intended (and how you run it is by definition how it is intended). Just as important, you need to see how the game runs with none of your intent present. Let the beta testers misread rules and take notes on what needs fixing.

I'd also test for play style as intended and optimized. What I mean by this is that the players should play the basic game you wrote. In D&D parlance that would be fighter, mage, healer, thief busting down dungeon doors. You need to make sure that your basic assumptions about the system work and that the system is fun to play when it's run correctly.

Testing for optimization/brokenness is another test entirely. Have the players try to break the system in half. In D&D terms, that's a party of Pun-Puns. If the broken game deviates too far from the ideal game, you need to tighten up your rules.

2012-06-04, 10:52 AM
In addition to what Valadil said, it might be good to have a mixed play group, some highly optimized and some less so. Hopefully the difference won't be huge. My current test group is like that, I have a really fluffy guy and a really optimized player and I'm trying to fine tune the system to make sure both have fun. So far the optimized guy is upset with how one trick pony his characters have to be to be really powerful.

2012-06-04, 11:42 AM
In addition to what Valadil said, it might be good to have a mixed play group, some highly optimized and some less so.

I hadn't considered that yet, but it's brilliant. Groups are rarely so homogenous as I was hoping they'd be for tests. It'd be interested to explicitly test the power gap between optmizers and casuals, and maybe come up with ways to make it fun for both sides.