Experimentation Culture

In this talk Oscar will share how Booking.com, the world’s leading accommodation website, enables product teams to have more than 1,000 tests running at any given time.

Oscar Carlsson, data scientist at Booking.com, described how experimentation became part of everyday product development there. Every change is tested against a hypothesis written before the test, in three parts: the theory behind the change, the metric that will show whether it works, and why it matters for the business. Users are split evenly between the old and the new behaviour, the run time is set in advance with a power calculation, and the result is judged with frequentist statistics, so that evidence decides rather than the highest-paid person’s opinion.

Since its first A/B test in 2005, Booking.com has built its own platform on the principles of democracy and decentralisation: experiments must be cheap, safe, easy, and reliable, so that any team, copywriters included, can run them without going through a central data-science team. More than 1,000 tests run at any time, on the website and apps, customer-service phone lines, partner tools, and internal systems, bug fixes included. Nine tests out of ten show no effect, so the culture asks people to be prepared to be wrong, and education and checks in the tool guard against looking at results early and writing hypotheses after the fact.