White Papers
Measure the Impact of Game Design Decisions by Starting with Player Outcomes

A few years ago, a couple of college students noticed 2 things: cafeteria food on their campus was not that great and other students we complaining that it takes too long to get their food. They felt they could do better.

They launched a startup and as their outcome, as their their flag on the horizon, they chose to help students to learn more efficiently by delivering delicious, home-cooked food directly to their door, saving them the time and the frustration that distracted them form their goal of learning.

They built, they collected feedback, they iterated… and they ended up with an alcohol delivery service. They changed their outcome instead of changing their design.

White Papers

A clear purpose saves us from frustration and empty wallet

It's been almost 2 years since I've made the switch from designing business and consumer apps to designing mobile games, and in that time I've found that one thing holds true, no matter the industry:

Design is the rendering of intent.

We can think of this as a fraction. There's a numerator and a denominator. If we want to change the result, we can do 2 things:

  • We can change the design (the numerator)
  • …or we can change the intent (the denominator)

Back in university, what we'd often do is that we would design a building and after the design was complete, we'd try to come up with out intent. Sometimes that worked well, other times not so well… but since our projects were never built, we never got to see the impact of our, admittedly flawed, process.

In the startup world, changing the intent is called a pivot. Startups tend to be small, to keep costs down but even then, a pivot tends to be expensive. It means reassessing every design decision we've made throughout the project's lifetime.

In game development, and especially during production, where we have teams of 20 - 60 people working on a project, changing the intent (pivoting) can get both expensive to the company and demoralizing to the team. Just ask any developer how much they love throwing out code.

White Papers

Identify and understand the high-level Player Outcomes

Whenever we design something, the one most important question we should be asking ourselves is:

If we do a great job, what does that look like for the player?

In the original Deus Ex game, Warren Spector, the game director, expressed his desired outcomes as commandments. Game pillars may be more familiar to some of us. One of Warren Spector's design commandments was Players do, NPCs watch. The player outcome of that commandment is that the player, and not just their character, is a constant part of the game world. We were there to do, not to observe. This in turn lead to very few and very short cutscenes in the game.

Contrast this with a game like Final Fantasy X which clocks in at 9 hours of cutscenes or compare it to most Hideo Kojima games. Hell, some game reviewers called Death Stranding A beautiful story burdened by a dull game. Now, whether the game is dull or not is not important to us, what is important is that Hideo Kojima's desired player outcome for Death Stranding was completely different from Warren Spector's desired player outcome for Deus Ex. Kojima wanted tell a story, Spector wanted players to be a part of one.

Now this is all very high level and it's likely that most of us looking at this article are not going to direct a game any time soon. However, it is still our duty as designers to ask for clear player outcomes from our Game Directors. How else are we going to know if we're taking the game in the right direction as a team… or working our way towards selling booze to college students?

Start with Player Outcomes in day-to-day design choices

We don't have to be Game Directors to set Player Outcomes. No matter what our role is, from game design to UI art to engineering, all we need to do is ask ourselves: If we do a great job, what does that look like for the player?

Now, I've heard it said that every good game designer starts out by stating how their design will impact the player. Unfortunately, when we're in the fires of production, we sometimes forget to state the obvious. Other times we confuse player outcomes with business outcomes:

  • Business outcome: Adding outfits for players to buy, will increase monetization.
  • Player outcome: Our player feels that their character is unique when they're in a match with other players.

Good, that's a start, but we need to dig deeper if we want to truly understand the player outcome well enough so that we can measure it later on, with our intended audience. The way we keep digging is by adding the words so that at the end of our initial outcome.

Player Outcome: Our player feels that their character is unique when they're in a match with other players, so that at the end of our initial outcome.

  • Player Outcome: Our player can showcase how much progress they've made in the game or how much they've won so far, so that
  • Player Outcome: Our player feels proud and accomplished when playing with others

Ok, this is getting interesting. So what we want as designers is for our players to feel proud of playing the game a lot… this is starting to sound like it's something we can tie into our business outcomes.

Player Outcome: Our player feels proud and accomplished when playing with others, so that

  • Business Outcome 1: Players return to the game every day, increasing our retention metrics. (you can even take this further by thinking out why retention metrics are important… e.g. so that our publisher feels more confident that our game will be successful, so that they decide to keep investing in our project, so that we have enough budget for marketing when we reach Soft Launch, so that…)
  • Business Outcome 2: Some players will want to skip the day to day effort and buy vanity items from the in-game store, so that we increase our monetization metrics. (…so that, we can keep working on this game we love)

Should players rely on memory or on snap judgements? Or on both?

Here's another example, these are 2 very different outcomes we can expect for players, though they do not necessarily exclude each other:

  • Player Outcome 1: Players need to rely on their memory.
  • Player Outcome 2: Players need to make snap judgements.

Let's take a look at Blizzard's Overwatch. More specifically at an Escort type of match. In an Escort match, one team has to move an object (the Payload) from one point of the map to another point, over a predetermined route.

The high level player outcome (memory or snap judgement) is a start but we need to dig deeper if we want to truly understand it well enough so that we can measure it later on. Let's again add so that at the end of our initial outcomes:

Player Outcome 1: Players need to rely on their memory, so that

  • Players understand that the goal of the game mode is to move the payload as quickly as possible, not to kill the enemy team.

Other Desirable Outcomes:

  • Players know where they need to move the payload.
  • Players know how to move the Payload.
  • Players know where the enemy team will come from.
  • Players know what alternate routes they can take through the map.

Player Outcome 2: Players need to make snap judgements, so that

  • Players can quickly determine when to stay next to the payload and when to chase the enemy.

Other Desirable Outcomes:

  • Players can quickly detect when an enemy is approaching
  • Players can easily distinguish between enemy characters, even when they are partly obscured
  • Players can detect and distinguish nearby enemy characters even is they can't see them

Obviously, we don't have to stop at the first step. Just like with Sakichi Toyoda's 5 Whys Technique, through which you ask Why 5 times to explore the cause and effect relationships of a specific problem, this approach works best if we keep identifying deeper reasons. To do that, we can keep adding so that at the end of our new player outcomes.

White Papers

How do we Measure the Player Outcome?

The thing about business metrics is that they are easy to measure in terms of quantitative data: Did the retention graph go up or down after we made the change? Did more people than our initial estimate buy the vanity item?

For player outcomes, analytics can certainly help if we're looking for things like the number of players who reach a certain level or the win/loss ratio for a specific fight. But when we're looking for what players understand and why players do something, we need a combination of:

  • Analytics
  • Playtests
  • Player Interviews (or surveys)

Let's look at one of our player outcomes:

Players understand that the goal of the game mode is to move the payload as quickly as possible, not to kill the enemy team.

I've chosen this particular outcome because, especially in low level leagues, players tend to run off after the enemy team and ignore the goal of the game, to move the payload. Another reason is because it annoys the hell out of me when it happens.

So what are we looking at? Here are some examples:

Analytics

  • Time spent by attackers next to the payload during a match
  • Time spent by defenders next to the payload during a match
  • Winning side (attackers or defenders)
  • Distance traveled by the payload

Playtests

  • Are the attackers pushing the payload forward?
  • When do they move away from the payload?
  • Do attackers chase after enemy players?
  • Are there strategic choke points for the defenders to take advantage of?
  • Are they taking advantage of those choke points?

Interviews

  • As an attacker, what do you need to do to win the game?
  • When playing as an attacker, how much time do you spend pushing the Payload?
  • When playing as an attacker, how important do you feel it is to chase down the enemy team?
  • Do you feel that there are times when the attacking team should not be pushing the payload?

These are just a few examples for a relatively specific set of design decisions. During the course of a game's development we'll be looking at a lot more things that we want to track, analyze or measure, to see if our intent is rendered properly.

Wrapping Up

At the start of this article, we mentioned a basic, working definition for what design is, created by Jared Spool:

Design is the rendering of intent

We should always see our designs through the lens of the player outcome because it's that player outcome that will tell us if our design elicits the reaction or behavior we're looking to achieve. In other words, it's the player outcome that tells us if our design is successful.

If you wanna learn more on the subject