Partial Blockout Experiments for a Two Sided Marketplace

Introduction

Booking.com is a two-sided marketplace, where we bring together customers looking for a place to stay when they are travelling, with our partners who offer a variety of accommodation options. In this context, partners are typically hotels or other accommodation owners.

On Booking.com, partners have a choice of a wide variety of products they can use to stand out and optimise how they capture their demand through Booking.com. We call these partner products.

Examples of partner products include the Genius programme (partners specifically target members of Booking.com’s loyalty programme), mobile rates (partners target customers browsing on their mobile with dedicated rates), Secret Deals (partners offer discounts only to logged in customers) and many more.

Figure 1. An example of a Genius accommodation on Booking.com. The Genius branding and discounted price help this partner stand out to members of the Genius program.

Fundamentally, partner products can only be successful if they deliver value both for our customers, but also for the partners that use them.

If these partner products drive a lot of value for our customers, but little value for the partners that use them, they will struggle to see meaningful adoption, limiting potential for success. Conversely, if they drive a lot of value for the partners that use them, but do not deliver much value for our customers, then they will not drive much business value even if they have good adoption, and may even lead to partners churning from other partner products, in favour of this less effective partner product. This is summarised in Figure 2.

Figure 2. How to reason about products, as a function of their impact for customers and partners. Impact for customers would typically be anything that makes it easier for them to find good accommodation — which could be measured as an increase in customer conversion. Impact for partners could be for example measured as the increase in revenue partners get when they use this product.

It is therefore critical that we have strong data and insights to understand the impact these products are having on both sides of our marketplace, and ideally we would gather these insights using the gold standard for causal inference — A/B experimentation.

Blockout Experiments

As is best practice at most large tech companies, we employ A/B experimentation extensively to validate all new changes and improvements we make to the website. At any time, we may have thousands of simultaneous A/B tests running, providing robust evidence upon which to make product decisions.

Blockout experiments are one of the tools in our experimentation toolbox. These are not experiments where we test something new, but instead experiments where we test removing an existing feature. They are very useful to understand the value that a specific feature or product is driving, which is often key for informing business strategy.

There are two common types of blockout experiment we run at Booking.com, each targeting a different side of our marketplace:

1. Customer blockouts (figure 3.A), where we remove the product or feature for a random selection of customers. These allow us to measure the impact of customer facing features of our website on metrics like customer conversion rate.
2. Partner blockouts (figure 3.B), where we remove the product or feature for a random selection of partners. These allow us to measure the impact of partner facing features of our website on metrics like the number of bookings they get.

However, as discussed previously, understanding both the partner and customer impact is critical when we are evaluating the performance of partner products.

These insights cannot be answered using a single customer or partner blockout experiment. These insights could be generated via other causal inference techniques, such as creating synthetic controls, but these typically require assumptions to be made, and those assumptions can be difficult to validate.

This led us to explore whether we could design an experiment which would allow us to directly measure the impact of our partner products on both sides of our marketplace.

Introducing the Partial Blockout

In a partial blockout experiment, for each customer the partner product of interest is removed (blocked) from a fixed percentage of the available partners. The partners that are blocked are randomised independently per customer, meaning a different, random set of partners are blocked for each customer.

This is in contrast to a customer blockout or partner blockout, which would block the product on all partners for 50% of customers, or on 50% of partners for all customers, respectively. The comparison between these setups is shown in Figure 3.

Figure 3. A comparison between a customer blockout (A), partner blockout (B) and partial blockout (C). Rows represent partners, columns represent customers.

To add a further dimension to the insights we can generate, and to provide us with more control over the amount of customers that are exposed to the blockout, we wrap the partial blockout inside one half of a standard customer split experiment. This leads us to the final setup for our partial blockout shown in Figure 4.

Figure 4. The design of the partial blockout experiment used at Booking.com. It combines the partial blockout inside the B group of a typical customer split experiment.

The partial blockout experiment has a lower cost, and is much less disruptive for both customers and partners, than either customer or partner facing blockout experiments. This is because the share of customer / partner combinations that are actually blocked can be kept quite low. Additionally, because the randomisation is in two dimensions, the product is never 100% removed for any single customer or partner, further reducing the disruption.

The two dimensional randomisation in a partial blockout allows us to extract an array of valuable insights about both sides of our marketplace simultaneously.

Unique insights from partial blockouts

Impact of increased partner adoption

The first case where a partial blockout really excels is in estimating the value of growing the adoption of a partner product. This is useful for making strategic decisions about which products to prioritise, and is a sensible metric to use on the x-axis of figure 2.

We can estimate this directly with a partial blockout, by comparing all customers who are not exposed to the partial blockout (group A) with all those who are exposed to it (group B). This is shown in figure 6.

Figure 6. To calculate the value of growing the adoption of a partner product we compare customers who are not exposed to the partial blockout (orange box) with customers who are exposed to the partial blockout (blue box).

If, for example, the partial blockout is set up to block 5% of partners per customer, then when comparing A and B we directly measure the impact of reducing our supply of that partner product by 5%.

To make this estimate independent of the share of the partner product we block out in the experiment, and therefore more comparable between different blockouts and products, we can approximate the marginal value of one additional partner adopting a product. This is done by exploiting the finite difference approximation of the gradient, and is shown in equation 1.

Equation 1. Estimating the marginal value of one additional partner adoption, dy/dN. Mathematically, this is defined as the gradient of y with respect to N. We derive the equation here from the finite difference approximation to the gradient.

It’s important to note that this is only a good approximation when the number of partners blocked is small. If the partial blockout is set up to block a large share of partners, then this will not yield a good approximation of the marginal value of one additional partner adoption.

Impact of a product for partners

The second unique insight we get from a partial blockout is an estimate of the value a product delivers for the partners that use it. We call this the partner impact of a product. This is the metric that would be on the y-axis of figure 2.

If we wanted to estimate the partner impact of a product for partner X, then ideally we would like to create a counterfactual where everything else on our platform is the same, except that partner X is not using the product. We can then compare the performance of partner X in the counterfactual vs in reality to estimate the partner impact.

A customer split experiment is not sufficient here, as it does not measure this counterfactual correctly. With a customer blockout, you are measuring a counterfactual where the partner product doesn’t exist at all, while what we want is a counterfactual where the partner product exists, but only partner X does not use it.

A partner split experiment does allow you to measure the correct counterfactual. However, it is in practice not feasible, since it would involve completely blocking a partner product for some partners, for the duration of the experiment, which could be extremely disruptive to their businesses.

A partial blockout avoids both of these problems. Since it only blocks the partner product for a small share of customers per partner, the disruption to a partner’s business is minimal. And it’s also possible to estimate the correct counterfactual, by comparing for example the bookings a partner gets when they are shown with the product, vs when the product is blocked (see figure 7).

Figure 7. To calculate the impact on the blue highlighted partner’s bookings, we compare the bookings that partner gets from customers who see it with the product (orange box) vs customers who see it with the product blocked (blue box). (Bookings here can be swapped for any key performance metric.)

This now estimates the correct counterfactual because each customer exposed to the partial blockout still sees the majority of partners with the product. Therefore, when you look at all the customers who saw this partner with the product blocked, you are measuring the performance of the partner in a world where the product still exists on other partners, but this partner does not offer it.

We can estimate the impact a product has on any metric, for a partner who is using that product, by simply comparing the total of the metric when the product is blocked vs when it is not, using equation 2 below.

Equation 2. Calculating the average uplift of any given metric, X, for a specific partner in the partial blockout experiment.

For example, if we want the understand the impact on bookings our product has for a specific partner, then we would calculate the total bookings that partner gets when the product is blocked, vs when it is not, and then use equation 2 to calculate the uplift.

To calculate confidence intervals on this estimate, we use bootstrapping, where we re-sample our dataset many times, each time repeating the calculation. You can find more information on bootstrapping here.

Important Caveats

While this method is very effective at calculating both the partner impact of a product, and the value of growing the adoption of that product, with minimal disruption to our partners, it does come with some important caveats.

1. The estimation of the marginal value of one additional partner adoption is only a good estimate when the share of partners blocked per customer is small. This is because it’s based on the first order finite difference estimation of the gradient.
2. The estimation of the partner impact of a product is also only valid when the share of partners blocked per customer is small. This is because the whole estimation is based on the customers in group B, who see a world with reduced supply of the partner product. Therefore, if the share of partners blocked is large, then average supply of the partner product is significantly reduced, and the estimates made will not be representative of the reality where we have full adoption.
3. Since the randomisation is performed on both the customer and the partner side, the same partner is exposed to both customers who see the partner blocked, and those that see it with the product. This can lead to interactions between the two groups. For example, when we are looking at metrics related to the number of bookings a partner gets, then if the partner sells out all their rooms to one group, there will be nothing left for the other group to book. This can lead to interaction effects between the groups and bias the results in supply constrained situations. However, when the partner is not supply constrained, the estimate is unbiased. For more details, see here.

Key Takeaways

Partial blockout experiments are an effective tool for evaluating and monitoring the performance of products in a two-sided marketplace, allowing the owners of the marketplace to reason about whether products are delivering sufficient value for both sides of their marketplace, and at a lower cost than running independent customer and partner split experiments.

They also provide the data foundation that can be used to provide valuable insights for the suppliers (in the case of Booking.com, our partners), to allow them to make better decisions on how they use the marketplace.

Acknowledgements

Thanks to my colleagues who helped deliver partial blockout experiments at Booking.com, especially Rajesh Shanmugam and Sina Kargar, as well as many others.

Thanks also to Ulf Schnabel, Stefanos Malandrakis and Katrina Visser who provided important feedback during the writing of this article.