Second WotC 5e Feedback Survey

[GWJ_DanyBoy]

Check it: http://dnd.wizards.com/articles/features/fifth-edition-feedback-survey

[pwykersotz]

I like their design goals on the ranger. I'm interested to see how this plays out. It's certainly setting the tone for future enhancements.

[Gwendol]

I like the reply regarding the results of the first survey. This allows for more options/customizations, rather than re-writes.

[DireSickFish]

I'm liking these surveys and there responses to them. Seems like they are really using all the information they can to expand on the game.

[Kurald Galain]

Actually, what I'd like to see is a third-party survey to see how 5E is doing compared to 4E and Pathfinder. I don't suppose that exists anywhere?

[CrusaderJoe]

Actually, what I'd like to see is a third-party survey to see how 5E is doing compared to 4E and Pathfinder. I don't suppose that exists anywhere?

Make sure you get someone who knows how to actually get decent data from a survey. These wotc surveys are terrible.

Let's say that 100% of the people think that "Tongue of Sun and Moon" is very dissatisfying. Wotc may assume all 100% of the people hate the feature (and thus shouldnt be oart of thr monk) when some may hate it because it comes on so late and some may hate it because it is useless... They aren't actually getting any data from these surveys.

They are getting What but not Why, to fix problems you must not only know what to fix but why it needs fixed.

One person may find Divine Smite to be very disatorfactory and another may find it satisfactory... And they have the exact same reason for choosing opposite satisfaction answers.

So yet again wotc isn't really getting reliable information.

[GWJ_DanyBoy]

Didn't we do this argument before? Hopefully we can avoid a simple repetition of the previous points. (http://www.giantitp.com/forums/showthread.php?395186-5e-Feedback-survey&highlight=survey)

[pwykersotz]

Make sure you get someone who knows how to actually get decent data from a survey. These wotc surveys are terrible.

Let's say that 100% of the people think that "Tongue of Sun and Moon" is very dissatisfying. Wotc may assume all 100% of the people hate the feature (and thus shouldnt be oart of thr monk) when some may hate it because it comes on so late and some may hate it because it is useless... They aren't actually getting any data from these surveys.

They are getting What but not Why, to fix problems you must not only know what to fix but why it needs fixed.

One person may find Divine Smite to be very disatorfactory and another may find it satisfactory... And they have the exact same reason for choosing opposite satisfaction answers.

So yet again wotc isn't really getting reliable information.

This seems to be the equivalent of saying that when you manage a network and all connected devices, you shouldn't poll for network alarms because sometimes programs crash, and that's completely unrelated, how dare they try to focus their information gathering on a wide net before organizing a team of specialists to look at it more closely!

Suffice it to say, while I agree that the survey by itself is not the only tool they need, it is a perfectly fine tool for certain applications.

[Galen]

I may be late to comment on this, but why is the Warforged so underwhelming? Keith Baker posted a very nice Warforged on his site, with two subraces (Soldier and Scout), why didn't they use that one?

[Galen]

Make sure you get someone who knows how to actually get decent data from a survey. These wotc surveys are terrible.

Let's say that 100% of the people think that "Tongue of Sun and Moon" is very dissatisfying. Wotc may assume all 100% of the people hate the feature (and thus shouldnt be oart of thr monk) when some may hate it because it comes on so late and some may hate it because it is useless... They aren't actually getting any data from these surveys.

They are getting What but not Why, to fix problems you must not only know what to fix but why it needs fixed.
This is absolutely true. They are looking at What not Why. It doesn't make the survey a bad one. Just aimed differently from what you'd possibly expect.

What they are doing with this survey is to cast as wide a net as possible to see where the problem areas are.

Give WotC some credit. I'm sure afterwards they will "zoom in" on the problem areas to determine what the problem is exactly. But this particular survey just tells them where to zoom in. Because apparently they don't have the resources to zoom in everywhere.

[GWJ_DanyBoy]

The warforged seemed fine to me, if a bit plain. I was more disappointed in some of the Shifter subraces.

[rollingForInit]

Considering their comments on the Ranger, it feels safe to assume that they're reading people comment's when they take the survey (there is a comments section, I assume I'm not the only one who got that?) and that they're browsing forums, reddit, etc to get a feel for what people think.

Having too many "why" might even be the best way to go with a huge survey. Pretty time-consuming to classify thousands and thousands of wall of texts.

[Gwendol]

Also, they just might think themselves the best suited to come up with the changes to address the "what"...

[MadBear]

They are getting What but not Why, to fix problems you must not only know what to fix but why it needs fixed.


You are correct about these survey's if they are intended to address both the what and why. However, you'll notice that they at least indicate that this isn't the case here:



We’ll start with an internal assessment mixed with feedback from our closed circle of testers. From there, we’ll work up some options and present them to you, most likely in the Unearthed Arcana column. That process allows us to determine if the track we’re on answers your concerns with the class.

So it seems that they're using play testers to look for the Why. Furthermore, they'll bring what they come up with back to the community to see if their answers properly address the problem.

Now, this might be too optimistic, but I'm willing to grant WOTC leeway since I loved the 5e product as a whole. With that said, it might be the case that if you're dissatisfied you're skeptical, which I can understand.

[CrusaderJoe]

You are correct about these survey's if they are intended to address both the what and why. However, you'll notice that they at least indicate that this isn't the case here:




So it seems that they're using play testers to look for the Why. Furthermore, they'll bring what they come up with back to the community to see if their answers properly address the problem.

Now, this might be too optimistic, but I'm willing to grant WOTC leeway since I loved the 5e product as a whole. With that said, it might be the case that if you're dissatisfied you're skeptical, which I can understand.

You are missing the point. All data they collect is faulty because of how they are gaining the data. You have to be able to interpret the data and you can only do that from the comment box data. All other data is faulty because you don't know what it means.

If I asked, would you be satisfied with spaghetti and meatballs as a meal tonight and all 100 people said "very dissatisfied". What does this mean? Is it because they are vegetarians an don't want the meatballs? Is it bacuse they don't want the carbs from the noodles? Is it because they don't like the taste of red sauce? Is it because they had so much spaghetti recently that they don't want anymore?

I have no way of interpreting the data. So if I get a group together to fix the dinner plan I really don't know which way to go because I didn't actually ask the right questions and I don't have reliable data.

It isn't about being skeptical, its about facts. Wotc's surveys arent actually doing anything. Everyone can be disatified about whatever, but unless they ask each person why they have no clue how to deal with the issue.

Now the part I am skeptical about now is if these surveys are even looked at. They can look at their own forums (or others) and find the issues with say the Ranger, and actually get real data from it in order to fix any problems people have. I wouldn't be surprised if these surveys are just all for show and they are getting their data solely on the forums.

Putting out surveys but following forum advice and data allows them to cheaply get data but appear and stand by " we are asking everyone" plan they have.

[Chronos]

Or another issue: Let's suppose I think that the classes are balanced poorly against each other (say, I think that the Warlock is more powerful than the Fighter, and steps too much on his toes). Is that an issue with the Warlock being too powerful, or with the Fighter being not powerful enough?

And just what does "satisfied" mean, anyway? Some people might interpret that as "is it powerful enough", while others interpret it as "is it the right amount of power". If I want my characters to be as powerful as possible, I might rate the most powerful class as having the highest satisfaction. But on the other hand, if I think that one class is much more powerful than all the others, I might think it makes the most sense to reduce the power of that one.

[DireSickFish]

Or another issue: Let's suppose I think that the classes are balanced poorly against each other (say, I think that the Warlock is more powerful than the Fighter, and steps too much on his toes). Is that an issue with the Warlock being too powerful, or with the Fighter being not powerful enough?

And just what does "satisfied" mean, anyway? Some people might interpret that as "is it powerful enough", while others interpret it as "is it the right amount of power". If I want my characters to be as powerful as possible, I might rate the most powerful class as having the highest satisfaction. But on the other hand, if I think that one class is much more powerful than all the others, I might think it makes the most sense to reduce the power of that one.

In your warlock example its going to depend on where you personally think the issue lies. If you want the fighter brought up to the warlocks level then you are dissatisfied with the fighter and what abilities you think give him "dead levels" or are boring or whatever. If its the other way around you knock the warlock and abilities you think are OP. If you think class design is terrible and they both need to be reigned into the middle you say you are dissatisfied with both.

Now you might be thinking that they won't know why you specifically are dissatisfied with the class and that is true (unless you post it int he comments section they provide). But it allows them to look at the classes people are dissatisfied with in more depth so the vocal minority are not driving things. There are forums like these and reddit where they can see these issues being brought up all the time. It isn't important in this survey that they know the why. It doesn't matter if you think it should be this way or that way. As far as system design goes they just need to know if you think things are good or bad. If enough people say a thing is bad (like they noted on the Ranger they will be taking more steps to offer fixes) THEN they will look at what the problem is and look at fixing it.

In essence its about where the problem is not what the problem is.

[MadBear]

You are missing the point. All data they collect is faulty because of how they are gaining the data. You have to be able to interpret the data and you can only do that from the comment box data. All other data is faulty because you don't know what it means.

If I asked, would you be satisfied with spaghetti and meatballs as a meal tonight and all 100 people said "very dissatisfied". What does this mean? Is it because they are vegetarians an don't want the meatballs? Is it bacuse they don't want the carbs from the noodles? Is it because they don't like the taste of red sauce? Is it because they had so much spaghetti recently that they don't want anymore?

I have no way of interpreting the data. So if I get a group together to fix the dinner plan I really don't know which way to go because I didn't actually ask the right questions and I don't have reliable data.

It isn't about being skeptical, its about facts. Wotc's surveys arent actually doing anything. Everyone can be disatified about whatever, but unless they ask each person why they have no clue how to deal with the issue.

Now the part I am skeptical about now is if these surveys are even looked at. They can look at their own forums (or others) and find the issues with say the Ranger, and actually get real data from it in order to fix any problems people have. I wouldn't be surprised if these surveys are just all for show and they are getting their data solely on the forums.

Putting out surveys but following forum advice and data allows them to cheaply get data but appear and stand by " we are asking everyone" plan they have.

I'm not missing the point at all, despite your claim.

The fact that they're playtesting means that they'll be using (or should be using as this is up for debate) a representative sample. If that's the case then the playtesters will see that the problem is the meatballs. In your analogy if 100% of the people who were dissatisfied were vegetarians then we ought to see a majority if not all of the testing group also be vegetarian. I'd also point out that you're using a faulty analogy anyway. The problem might be that one group is vegetarian, one group despise pasta, the other hates marinara, and so on. A representative sample group will crop up with these problems and can show the issue there proportionally. Are you really suggesting that their playtesters are so vastly different from the people who took the survey that they won't have the same issues come up?

The forums are an ok place to take sample data from, but it by no means is an end unto itself. What % of people who play D&D use the forum? 50%/10%/1%? We don't even know if we're a representative sample of people who play the game. Using a survey to collect some initial data is perfectly fine.

[Galen]

You are missing the point. All data they collect is faulty because of how they are gaining the data. You have to be able to interpret the data and you can only do that from the comment box data. All other data is faulty because you don't know what it means.

If I asked, would you be satisfied with spaghetti and meatballs as a meal tonight and all 100 people said "very dissatisfied". What does this mean? Is it because they are vegetarians an don't want the meatballs? Is it bacuse they don't want the carbs from the noodles? Is it because they don't like the taste of red sauce? Is it because they had so much spaghetti recently that they don't want anymore?
No, you are missing they point. Spaghetti and Meatballs is not the only meal on the menu. What they are really asking is:

1. Are you satisfied with the Chicken Alfredo? [98% say yes, 2% say no]
2. Are you satisfied with the Pork Chops? [95% say yes, 5% say no]
3. Are you satisfied with the Buffalo Wings? [97% say yes, 3% say no]
4. Are you satisfied with the Spaghetti and Meatballs? [21% say yes, 79% say no]

Aha. So, at the cost of a simple survey, we have learned that the Chicken Alfredo, Pork Chops, and Buffalo Wings are alright, and do not require further corrective action. Spaghetti and Meatballs, we have learned, seems to require corrective action. What action exactly? We don't know yet. But we will find out by investigating further.

Such survey is very efficient in finding out which item(s), out of many, require corrective action. Admittedly, it's not very good at finding out what the corrective action is. But you'd be naive if you'd think this survey is the only tool at their disposal.

[Gwendol]

It isn't about being skeptical, its about facts. Wotc's surveys arent actually doing anything. Everyone can be disatified about whatever, but unless they ask each person why they have no clue how to deal with the issue.

Putting out surveys but following forum advice and data allows them to cheaply get data but appear and stand by " we are asking everyone" plan they have.

Of course the surveys have a purpose: it is a tool to find out how the product is doing: what players are happy with and what they are not so happy with. They've had the open playtest to hone their skills in survey evaluation, and develop a working process to carry the work onward. Active forum participants are likely a vocal minority in the whole of the D&D community, so listening to them/us (only) would likely be a mistake.

[jkat718]

The way I see it, WotC probably has people looking at forums and message boards and the like, so they know what people's complaints are. The surveys are just to get a measure of how important the changes are too people. Just my 2ep*.

*Electrum is the worst, forget copper.

[Vogonjeltz]

Now the part I am skeptical about now is if these surveys are even looked at. They can look at their own forums (or others) and find the issues with say the Ranger, and actually get real data from it in order to fix any problems people have. I wouldn't be surprised if these surveys are just all for show and they are getting their data solely on the forums.

There are only so many people working on the development of a game and they have only so much time (man hours) to devote to any given task. Online polling is basically cost free and allows them to note where the most widely encountered problems are and focus their efforts.

Using a survey is intrinsically more efficient than trying to manually collect data from a forum, all the more so when you are trying to establish a base-line of what is and what isn't working.

If everyone and their dog agrees that Clerics and Sorcerers are Satisfactory, nobody wastes precious time looking into those classes.


And just what does "satisfied" mean, anyway? Some people might interpret that as "is it powerful enough", while others interpret it as "is it the right amount of power". If I want my characters to be as powerful as possible, I might rate the most powerful class as having the highest satisfaction. But on the other hand, if I think that one class is much more powerful than all the others, I might think it makes the most sense to reduce the power of that one.

You are still asking "why". Satisfied means: Contented; Pleased. The "why" doesn't matter.

[Chronos]

But of course it matters. If you don't know why someone isn't satisfied, you can't know how to fix it.

[pwykersotz]

But of course it matters. If you don't know why someone isn't satisfied, you can't know how to fix it.

It doesn't matter why they're SATISFIED. They're good, change nothing unless future results indicate differently.

It matters why they're not satisfied. Hence their focus group on the ranger.

It's pretty basic.

[Vogonjeltz]

But of course it matters. If you don't know why someone isn't satisfied, you can't know how to fix it.

The second sentence is true, and it's also the second step AFTER problem area identification takes place.

Step 1: Identify that there is a perceived problem (this is what the survey is for).
Step 2: Glean the specifics of perceived problems.

Trying to skip directly to step 2 is a wild goose chase.

[Giant2005]

You can count me among the number that hated the way they wrote/conducted the survey.
I have no idea if they are going to interpret my responses the way I want them to because I couldn't give them enough information to make an accurate assessment.
How much someone likes something is kind of irrelevant.
I might dislike something because it is broken and OP, yet someone else might like it because it is broken and OP. The results themselves have no way of indicating whether something is broken or not.

[asorel]

You can count me among the number that hated the way they wrote/conducted the survey.
I have no idea if they are going to interpret my responses the way I want them to because I couldn't give them enough information to make an accurate assessment.
How much someone likes something is kind of irrelevant.
I might dislike something because it is broken and OP, yet someone else might like it because it is broken and OP. The results themselves have no way of indicating whether something is broken or not.

The intent of the survey is to make a quantitative breadth-first search, rather than a qualitative depth-first search. I have no doubt that the developers haven't given the fora at least a cursory glance before choosing how to parse this data. They are likely to follow this up with a more in-depth survey regarding the areas with which people had the most issues. Even if they didn't, it's already been specified that the fixes will supplement the existing classes and features, not rewrite them. Finally, historically the developers have been to an extent open with 5e's design process, with the 18 months or so of playtesting that happened beforehand, and the Unearthed Arcana articles being published every month. I wouldn't be at all surprised if we receive some sort of insight into how they implement these fixes well before the decision is final.

[Giant2005]

The intent of the survey is to make a quantitative breadth-first search, rather than a qualitative depth-first search.

But that is the point - there is nothing quantitative about it.
I genuinely had no idea how to answer - I am a player so OP classes are great for me but I'd rather a balanced game. So how exactly do I rate that? Do I rate it from the perspective of a player (like the OP crap) or do I rate it from the perspective of what actually makes a better game (dislike anything unbalanced regardless of what axis it falls on).
Because of using absurd metrics like "like", the survey loses any real value as the results will be skewed by people answering based on one perspective and others answering based on another. Two people trying to make the same statements may end up with very different survey results regardless of their intentions.

[TheOOB]

But that is the point - there is nothing quantitative about it.
I genuinely had no idea how to answer - I am a player so OP classes are great for me but I'd rather a balanced game. So how exactly do I rate that? Do I rate it from the perspective of a player (like the OP crap) or do I rate it from the perspective of what actually makes a better game (dislike anything unbalanced regardless of what axis it falls on).
Because of using absurd metrics like "like", the survey loses any real value as the results will be skewed by people answering based on one perspective and others answering based on another. Two people trying to make the same statements may end up with very different survey results regardless of their intentions.

I think you're misunderstanding what the survey is about. All they care about for the survey is how much players like or dislike certain things. Lots of people disliked the ranger, so they are going to take a look at it and try to make it better, using the survey to see what aspects most need attention.