Experimentation denial stages (and how to navigate them)
Experimentation (or A/B testing) is a fantastic tool that gives teams confidence in the impact of their improvements and changes.
But it comes at a cost: on top of a technical setup and basic statistics training, adopting experimentation requires a significant mindset shift and acceptance of the idea that you’re more often wrong than right. This realization is uncomfortable, leading many people to resist experimenting.
Organizations trying to overcome this resistance typically go through the same stages:
1. Who needs experimentation?
“We know what we’re doing, we’re the experts, and we also do usability testing.” Nothing helps better at this stage than running first tests and letting people see how intuition or common sense can be wrong. Get them involved in predicting the outcome of the test: if they guessed right, it’d give them a nice shot of dopamine. Guessing it wrong will make them question their intuition and assumptions.
2. It’s a waste of development time.
“Now we’re shipping new features twice as slow as before.”
Would the team rather ship a lot of stuff with unknown value and impact?
Help team members to see the value by looking at data together. Solving mysteries of user behavior change (or lack thereof) is as contagious for developers as it is for product people.
3. Experimentation is “a step in a feature release process.”
Experimentation is accepted as a checkmark along the way to the release of a feature. PMs and designers find ways to rationalize inconclusive results without looking at what data experiment gives them. At this stage, teams don’t see experimentation as a source of insights, so putting time and work into learning to pause and reflect pays off.
4. Running experiments for the sake of running more experiments.
At this point, to address people’s reluctance, leadership introduces the number of experiments as a KPI. Teams, forced to optimize for it, do just that — run experiments for the sake of running experiments.
This approach is known as throwing spaghetti on the wall (you do random things and see what sticks). Adding restrictions like following a specific hypothesis template helps but doesn’t solve the underlying issues: teams aren’t that skilled in discovery (which product leadership should address), and there are only that many reasonable small changes that a team can do in an area.
While lowering the quality of experiments, this stage also does some good by making people comfortable running many experiments.
5. Experimentation dictatorship.
Pushed too hard, teams give up their autonomy to the experiment tool and view it as an ultimate decision-making mechanism (which it’s not). This era is survived by long-term strategic thinking and bringing more qualitative insights to complement experimentation data.
6. Acceptance
Experimentation is seen for what it is — a healthy way to validate your assumptions and inform your decisions. It’s neither a blocker for product work nor a decision-making mechanism that decides for the team.
One of the biggest obstacles to good experimentation is the “ship a feature and move on” product mindset. The process of overcoming it is not straightforward and will cause side effects like “throwing spaghetti on the wall” experimentation. Healthy experimentation culture requires ongoing work and education to both see all the benefits of experimentation and learn to use it in combination with discovery and other validation methods.
