Experimentation denial stages (and how to navigate them)

Experimentation (or A/B testing) is a fantastic tool that gives teams confidence in the impact of their improvements and changes.

But it comes at a cost: on top of a technical setup and basic statistics training, adopting experimentation requires a significant mindset shift and acceptance of the idea that you’re more often wrong than right. This realization is uncomfortable, leading many people to resist experimenting.

Organizations trying to overcome this resistance typically go through the same stages:

1. Who needs experimentation?

2. It’s a waste of development time.

Ship faster

Would the team rather ship a lot of stuff with unknown value and impact?

Help team members to see the value by looking at data together. Solving mysteries of user behavior change (or lack thereof) is as contagious for developers as it is for product people.

3. Experimentation is “a step in a feature release process.”

4. Running experiments for the sake of running more experiments.

This approach is known as throwing spaghetti on the wall (you do random things and see what sticks). Adding restrictions like following a specific hypothesis template helps but doesn’t solve the underlying issues: teams aren’t that skilled in discovery (which product leadership should address), and there are only that many reasonable small changes that a team can do in an area.

While lowering the quality of experiments, this stage also does some good by making people comfortable running many experiments.

5. Experimentation dictatorship.

6. Acceptance

One of the biggest obstacles to good experimentation is the “ship a feature and move on” product mindset. The process of overcoming it is not straightforward and will cause side effects like “throwing spaghetti on the wall” experimentation. Healthy experimentation culture requires ongoing work and education to both see all the benefits of experimentation and learn to use it in combination with discovery and other validation methods.

--

--

Combining data and psychology for product and design decision-making | Head of Design at DeepL | elenaborisova.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Elena Borisova

Combining data and psychology for product and design decision-making | Head of Design at DeepL | elenaborisova.com