Beyond the Winner: The Case for Optimization of Game A/B Tests
The value of optimization
When it comes to A/B tests, here is one critical thing big game studios do that indie developers and startups frequently overlook: they optimize the test winner.
If you successfully implement a gold-nugget feature in your game that tests at a nice +26% uplift in player attempts against the control group, you might be tempted to do your checks, declare a winner, roll out the feature, and blow the horn for a round of applause. But gold concentrates in localized pockets. If you find some, keep digging, because otherwise, you might be leaving even bigger opportunities untapped.
Based on my direct experience, optimizing a winning feature can easily increase its uplift by half, and in some cases, even double it. When successful, optimization has the potential to not just raise the baseline of the existing uplift; instead, it can change how the winning series behaves over time, making it more stable and robust.
Caveats and risks
There are indeed some caveats and pragmatic reasons explaining why less resourceful teams tend to skip optimization and run with the loot. Time constraints and the risk of haphazard tweaks are definitely two important factors. In a mobile game, even a simple feature can be tweaked in many ways, but only a small subset of these changes will actually result in a performance improvement.
Moreover, without clear test ownership, every member of the team will have a strong opinion about what could work. Instead of relying on a well-designed optimization strategy, these premises can lead to a trial-and-error discovery phase that can waste a lot of time and ultimately lead nowhere.
Guidelines for small teams
So, how can small teams carry out an effective and efficient optimization of a winning test feature?
The answer is to stand on the shoulders of giants. Winning streaks, racing competitions, welcome-back rewards, gacha systems, and special levels are some of the proven features that are part of the playbook that experimentation and product teams from big companies have learned to design and optimize extremely well, thanks to their resources and player volume.
From this perspective, each feature in the playbook can be optimized tackling three different aspects:
- Presentation: How the feature is introduced to players, and which visual elements underpin the correct comprehension
- Mechanics: How the mechanisms and flow of the feature work
- Behavior: The behavioral drivers that the features taps into and should effectively activate
A winner can still have weak spots across these elements, weaknesses that knowledge of the playbook can identify already in the stage of featured design stage or quickly uncover during optimization with specific, targeted data analysis.