Beyond the Winner: The Case for Optimization of Game A/B Tests
The value of optimization
When it comes to A/B tests, here is one critical thing big game studios do that indie developers and startups frequently overlook: they optimize the test winner.
If you successfully implement a gold-nugget feature in your game that tests at a nice +26% uplift in player attempts against the control group, you might be tempted to do your checks, declare a winner, roll out the feature, and blow the horn for a round of applause. But gold concentrates in localized pockets. If you find some, keep digging, because otherwise, you might be leaving even bigger opportunities untapped.
Based on my direct experience, optimizing a winning feature can easily increase its uplift by half, and in some cases, even double it. When successful, optimization has the potential to not just raise the baseline of the existing uplift; instead, it can change how the winning series behaves over time, making it more stable and robust.
Caveats and risks
There are indeed some caveats and pragmatic reasons explaining why less resourceful teams tend to skip optimization and run with the loot. Time constraints and the risk of haphazard tweaks are definitely two important factors.
In a mobile game, even a simple feature can be tweaked in many ways, but only a small subset of these changes will actually result in a performance improvement.Moreover, without clear test ownership, every member of the team will have a strong opinion about what could work.
Instead of relying on a well-designed optimization strategy, these premises can lead to a trial-and-error discovery phase that can waste a lot of time and ultimately lead nowhere.
Guidelines for small teams
So, how can small teams carry out an effective and efficient optimization of a winning test feature?
The answer is to stand on the shoulders of giants. Winning streaks, racing competitions, welcome-back rewards, gacha systems, and special levels are some of the proven features that are part of the playbook that experimentation and product teams from big companies have learned to design and optimize extremely well, thanks to their resources and player volume.
From this perspective, each feature in the playbook can be optimized tackling three different aspects:
- Presentation: How the feature is introduced to players, and which visual elements underpin the correct comprehension.
- Mechanics: How the mechanisms and flow of the feature work.
- Incentives: The behavioral drivers that the features taps into and should effectively activate.
A winner can still have weak spots across these elements, but a detailed understanding of its core tenets can help identify them during feature design, or quickly uncover opportunities during optimization through targeted data analysis.
Summing up
The value of experimentation doesn’t stop at discovering what works. Usually there is still considerable value behind maximizing the potential of test winners. For resource-strapped teams, the temptation to “run with the loot” is real, but a structured optimization phase can unlock considerable value and make a feature really shine. By evaluating your successful features through the triple lens of UI, mechanics, and incentives based on industry best practices, you can systematically design better features and identify key opportunities for optimization.