How to grow your game through experimentation

• The Experimentation Group was created last year by former EA and King veterans, Tom Storr, Phillip Black, and David Nelson
• The consultancy groups goals were to help teams for free if they we then able to share their results publicly so others could also learn from the process

Last year, industry veterans Tom Storr, Phillip Black, and David Nelson came together to create a new consultancy project called The Experimentation Group. With years of experience at EA and King, the three set out to offer services to studios seeking guidance free of charge.

Here, The Experimentation Group’s Tom Storr details some of the lessons learned from the projects that the team has helped on over the last eight months, and how you can apply them to your own project.

From King to TapNation via the legendary Lawson at Heseri Games, we’ve worked in experimentation for over a decade.

Over that time, we’ve consistently seen how, when applied correctly, experimentation can unlock huge growth for games of any size. This article summarises what we’ve learnt across these experiences.

The starting point has been consistent: A team has been working on a game for a while,  performance isn’t where they would like (maybe it’s fallen off, maybe it was never there).  They’ve tried a bunch of strategies to turn things around, but nothing has had the desired effect.

Believing in their game, they’ve then got in touch.

Summarising learnings across every project we’ve worked on, this is our advice if you’re in a similar spot and thinking of trying it for yourself.

1. Operate within a structure

Experimentation isn’t about taking wild bets and hoping for the best.

You should be clear about what you’re trying to achieve and why you think it will work with your player base, then methodically work towards that goal. This means being clear on the metric you’re trying to move and your biggest hypotheses.

What metric do you want to move?

Money, obviously. However, we have to go deeper than that and carefully consider each candidate's implications.

It pays to target a level you can impact directly. Let’s consider average revenue per user (ARPU). This could be achieved by filling your game with ads. While ARPU may grow, you’ll probably also kill retention, which isn’t what anyone wants.

What about average revenue per paying user (ARPPU)? This might reduce the retention risk, as you’re targeting players who have spent already. But it also means a small target, making it harder to measure reliably.  It’s also a really precious segment, one you need to be careful with.

Neither sound like a great option. But they’re not necessarily wrong. The point is to carefully consider the implications of your decision.

We’ve found the most common starting point is early retention. Often, day 1 (D1) to day 3 (D3) retention is low, which translates to low retention across the curve. This also impacts the ability to spend profitability on user acquisition (UA).  

Working on D1 to D3 has a number of benefits:

Fast answers:

• Maximum players exposed to the treatment
• The treatment is visible early in the experience by definition

Potential for trickle-down effects:

• Lifting D1 to D3 retention can lift later parts of the curve
• Learnings could be applied to later parts of the experience

Relatively low risk:

• These players are fresh to the game, and there is less risk if things go wrong
• You may be able to test in cheaper markets, again protecting your best assets

As a side note, we’ve often started by removing things, balancing levels, or reordering when features are introduced. So, it's relatively light work, which is also helpful.

What’s your strongest hypothesis?

This is so important. Don’t be tempted by anything other than your best theory about what’s going to move your target metric:

If you’re right:

• It will be easy to see, players will respond quickly and clearly
• Victorious, you can immediately dig deeper into your understanding, testing ways to make the most of it

If not, you will have saved yourself a lot of time and energy

• This might sound counterintuitive, but a negative result is amazing
• You’ve removed a misunderstanding that would have been your biggest blocker to progress
• You can now move on to your next biggest hypothesis until you find what truly moves your players

2. Maintain a healthy tension between data and creativity

The best teams balance data and creativity to generate fresh ideas that are grounded in an understanding of their player base.

Focusing too much on data leads to ever-decreasing gains. Teams are inwardly focused. New features are variations on a theme. They will never be enough to prompt a significant change in trajectory.

Unbridled creativity reduces your roadmap to a lottery. Features with no reference to player behaviour are unlikely to work because they have no basis in truth. Creativity needs to be within the frame of your player base to be effective.

Thinking broadly in a way that relates to user data balances these two needs.  

Take inspiration from other games, other apps, the real world, or an experiment you’ve read about, but tie it back to your understanding (or beliefs if you’re really starting from scratch) about your players.

There is no harm in drawing inspiration from other experiences, as long as you have a clear reason for doing so. Your goal is to learn, so if you think your players would respond to a competitive, short-term goal and you know a live op that has a great one, try it and see what happens.  

The true value is understanding whether your players responded to the treatment

If they did, you can start to innovate on it yourself.

To ensure your experiments are based on data, make it a requirement to add an explanation to the hypothesis.

For example:

A live op where players race to complete the next five levels will increase session length because experiment X showed our players love to compete.

This simple change will help filter out fun ideas that, even if successful, won’t help you build your understanding.

3. Move fast

Finally, move fast and test often.

This is the biggest differentiator between teams we’ve seen make progress with experimentation and those who haven’t.

The true value of experimentation is building an understanding of your players.  

The more chances you have to learn, the smarter you will become and the better decisions you will make. That is your competitive advantage.  

So be concerned about experiments that will take a month or more to ship. Home runs are rare. Spending four weeks building and two weeks (plus) waiting for data for a flat or unclear result is really demotivating (this is another reason for starting with your biggest hypothesis; it is most likely the result has value).

You should aim to get your first swing out the door in a week or two. Keep it simple in terms of cells (control and one test cell only) so you get a result in a matter of days.  Review the data, discuss what you think it means and agree on which experiment you believe will have the best impact next.

Each time you’ll learn something new and make a step forward.  

We’ve found getting results back is a real moment for teams. Having data allows the discussion to move on from hypotheticals and hazy understandings. It democratises debate across the team, taking it from “what if” discussions that opinionated individuals can dominate to creativity bound in understanding everyone can contribute to. It’s an exciting moment.

Summary:

I work in experimentation because I love it. Every experiment is fun. I see that in the teams we work with. The outcome is rarely as you expect, but the learning always takes us forward.

So, if you want to give it a go, remember:

1. Operate within a structure

2. Maintain a healthy tension between data and creativity

3. Move fast

4. Have fun!

Edited by Paige Cook