How to speed up test results?(and avoid waiting a year)

---

Summary

To follow A/B Testing best practices, we can’t speed up results.
If waiting a year isn’t an option, we can use a proxy. However, we must accept more uncertainties in the results.

---

Let’s take an example:

I sell subscriptions due to renewal every 12 months. I have two projects aimed at improving the retention rate (RR%):

• The first is to make additional payment attempts if the previous one doesn’t work.
• The second is to offer PayPal in the check-out because it’s a method of payment known to offer better retention.

I need to validate these projects do improve my business. I will use either the existing reporting or the A/B Testing method.

1. Collecting the results

Project #1 – multiple attempts: immediate results ⏱️

I set up a test with a control group. When a subscription payment is due, I make a first payment attempt. If the attempt fails, I try again 24 hours later.
First attempts start on day 1, re-tries start on day 2. I can compare the performance of the two groups as soon as day 2 ends. The results are almost immediate. If successful, I can generalize the practice of multiple billing attempts and materialize the benefits on the retention rate immediately.

Project #2 – new payment method: deferred results 📅

I added PayPal as a new method of payment on January 1st of year N for new customers (acquisition). The first renewals will start a couple of weeks before the anniversary date of the subscription, i.e. mid-December of year N.
After a few weeks of renewal, I can compare the retention rate of PayPal against Visa, Mastercard and Amex customers. In this situation, PayPal being recently added, the number of renewal transactions made will be very low compared to Visa/MC transactions. I must wait for enough transactions to be statistically significant.

Both projects meet the goal of improving the retention rate, but the time to obtain results is very different. I don’t necessarily have time, desire or luxury to wait a year to know if the result is positive.

2. Finding shortcuts ⚡️

The benefits on the retention rate only materialize after the renewal. To save time, I will first try to test changes at times closer to the renewal.

Consider the following example:

For my project #2 (adding PayPal), I will offer customers to update their payment information one or two months before renewal. To increase efficiency, I will first address customers whose cards will be expired when I’ll start billing attempts. My communication will highlight the new PayPal option. Then, I will perform comparisons of retention rates by method of payment, only on this cohort of customers.

This test compares the “Billing Success” of each payment method. It doesn’t allow us to affirm that purchases with PayPal have a better retention rate. But since it’s unlikely that the choice of the initial payment method has any influence on the Billing Success, we can be satisfied with this approximation.

3. 🐫 Consider a proxy 🐪

Before continuing, I specify that a subscription business has 3 macro KPIs:

• Number of subscribers
• Retention rate (RR%)
• Average revenue per customer (ARPU)

All other KPIs are micro KPIs. They all contribute to influencing one or more macro KPIs.

Sometimes, it isn’t possible to find shortcuts, and for which waiting 12 or 15 months isn’t an option. It’s then necessary to consider a proxy.

A proxy of my KPI is another micro KPI that has a similar influence to my KPI on the macro KPI. However, I will be able to measure the proxy more quickly.

Let’s take an example :

the upgrade rate is a micro KPI that influences the product mix. If it increases, my average price increases. This micro KPI influences the ARPU which is one of my macro KPIs.

Concretely: I measure a variation of the upgrade rate. I can predict my new product mix. Knowing the average retention rates per product and the prices, I will be able to predict the variation of ARPU.

4. How to find a proxy 🔎

I find a proxy using my reporting system. Starting from the macro KPI to influence, I look for micro KPIs that have a strong correlation with the macro KPIs. When I have identified one, I look for the conditions of influence. Then I perform a test whose goal is to influence this proxy.

Let’s consider example #1 :

I want to influence the retention rate. I note that product A has a retention rate 10 points lower than product B. Rather than testing different product mixes in Acquisition, I will test the sale of upgrades in the middle of the lifecycle.

As with the previous method, I only target subscribers close to renewal. As soon as I get the test results, I validate that the macro KPI and the micro KPI have both been modified in similar proportions to what the reporting system indicated. If this is the case, my proxy is valid and I can extrapolate.

Here’s example #2 :

My project is to improve the retention rate (RR%) by encouraging customers to use my app 365 days in a row.

I identify “time spent on the app” as an eligible proxy. My reporting system tells me that customers who use my app less than an hour per week have a RR% 5 points lower than those who use it between 1 hour and 2 hours.

I launch an A/B test, encouraging customers to use the app every day. I make sure that the weekly usage time reaches 1 to 2 hours in order to find myself in the same conditions. Only customers close to renewal are targeted.

When measuring results, I take care to validate that daily use corresponds to a time spent of 1 to 2 hours and that the increase in RR% is also in a 5 points range. If this is the case, I consider that my proxy is valid ✅ and I can extrapolate the results.

Key takeaways

1. The reporting system is my best friend 😍 to find alternative solutions and save time. ⏱️
2. During a test, if the first results take several months to come, then it’s advisable to consider a simulation of immediate result for validation. We don’t want to wait 3 months to find out that the KPI is not measured correctly.
3. 🚧 Caution 🚧 : If I launch a test and it takes me 12 months to realize the change is harmful. Immediately, I cancel it. However, I will take another 12 months to return to the previous performance. This is why a test must be short with quick results.