Experimentation guide for product designers and managers

0. Do we really need experimentation?

Experimentation, or A/B testing, is the most reliable way to see a cause-effect relationship between your product change and your metric change. Pre-post analysis might work but it severely limits the number of changes that can be done in a given time and is influenced by seasonality. Organizations that want to measure the impact of improvements introduced by product and development teams have to adopt A/B testing.

Experimentation as a filter: what we think are the good ideas and what are the actually good ideas
A simplistic understanding of experimentation
Observation->Hypothesis->Experiment->Results->Observation cycle
Experimentation learning cycle: results of previous experiments inspire new experiments

1. How to experiment well

Experimenting well requires a shift of mindset and is not easy. Experimentation proves teams wrong more times than is comfortable. Things that seemed like “obvious user experience improvements” don’t have the expected impact. And instead of moving on to the next item on the roadmap, a team has to iterate, which is frustrating and time-consuming. So many people feel resistance to experimentation.

2. How to write a good hypothesis

A hypothesis is a foundation for an experiment. It ties together the impact you expect to see from your change and why you think this will be the case.

Weak hypothesis

If we do [a treatment], we’ll see [change in metric]

Weak hypothesis lacks a foundation of observations or prior knowledge

Strong hypothesis

Based on [research, data, past experiments, psychology principles etc.] we believe that

Summing things up

Experimentation is an essential tool for measuring the impact of product changes, but it’s difficult to adopt. It requires not only a technical setup but also a mindset shift.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Elena Borisova

Elena Borisova

167 Followers

Product, data, decision-making, philosophy| Learning from everything that went wrong | Digital Product Designer | elenaborisova.com