CRM A/B Test Plan
- Elleni, Anna
- Saidy, Pierre
A/B testing, also known as split testing, is a crucial method for optimizing CRM strategy. By comparing two versions of a webpage, email, or other CRM element to see which performs better, we can make data-driven decisions that enhance customer engagement and satisfaction. This document outlines how we build the A/B testing calendar for each markets' CRM strategy and proposes several A/B tests and summarizes related results.
Creating an A/B Testing Calendar
Define Objectives
We have started by defining the key objectives and questions we would like to have answered within the next few months, according to the market specific priorities and status
Identify Key Metrics
Not all the A/B tests might be necessarily measured based on sales impact, but the overall objective behind them should be to optimize our CRM performances. Therefore, we should have some north star metrics (sales, measured via the global control group methodology) and along with key metrics, that could vary by A/B tests. For example:
Email open rates
Click-through rates (CTR)
Conversion rates
·Customer retention rates
· LTV
Prioritize Tests
Focus on tests that align closely with objectives and have the potential for significant impact. Assign a priority level to each test (e.g., high, medium, low) ensuring we are not running multiple tests at the same time to have clean measurement
Create a Testing Schedule
Develop a schedule that outlines when and how each test will be conducted. A sample monthly calendar might look like this:
· Week 1: Test Subject Lines for Email Campaigns
· Week 2: Test Call-to-Action (CTA) Buttons on Landing Pages
· Week 3: Test Email Content Variations
· Week 4: Test Customer Journey Flows
Ensure to leave sufficient time for data collection and analysis between tests.
Q4 - Markets' A/B Testing Calendar
Burger King Switzerland
Opportunities
Increase loyalty sales
Increase contactable sales
Increase CRM Impact
A/B tests plan
Objective | Status | Test | Plan | Results Con |
|---|---|---|---|---|
Contactable Sales | Only 76% of the contactable base is actually being reached with communications | Target users who are opted in to push notifications but don’t have push turned on from the device  McD DE | 24/10/2024 |
|
Contactable Sales | Only 76% of the contactable base is actually being reached with communications | A/B test when to prompt the user (when they download the app vs when they they redeem their first coupon) |
|
|
Contactable Sales | 52% of the new sign ups are opted in 
| Test different copy to incentivize opt in (e.g. want to unlock more coupons?) | 24/10/2024 | 07/11/24: 4.1% opt in rate for user that saw the IAM |
CRM Impact |
|
|
|
|
CRM Impact | Some of the journeys are underperforming.
|
|
|
|
CRM Impact | We don’t have a ready to activate journey to react to McD initiatives
| Test time bomb coupons |
|
|
CRM Impact | We have 90K users (26% of our email contactable base) who did not open an email in the last 180 days https://app.eu.amplitude.com/analytics/burgerkingch/cohort/e-0a4iub3m | Test a more aggressive copy to reactivate lapsed customers (test different copy variants) |
|
|
Loyalty Sales | BKCH is behind average in terms of loyalty sales compared to EMEA main markets Power BI - September YTD | Test refer-a-friend mechanics to leverage network effect to create incentives to join the program |
|
|
Loyalty Sales | Contribute to increase average check through second ticket
| Test dessert coupon push, 5, 10, and 15 minutes post transaction
| 24/10/2024 | 07/11/24: Good result on all variants with the 5 min one outperforming all. |
Burger King Netherlands
Opportunities
A/B tests plan
Objective | Status | Test | Plan | Results Con |
|---|---|---|---|---|
Contactable Sales | Only 33% of the loyalty base is opted-in for communications |
|
|
|
Contactable Sales | Only 63% of the contactable base is actually being reached with communications |
|
|
|
Contactable Sales | 40% of the new sign ups are opted in. The curve has been slowly decreasing since May.
| Test different copy to incentivize opt in (e.g. want to unlock more coupons?) |
|
|
CRM Impact |
|
|
|
|
CRM Impact | We don’t have a ready to activate journey to react to McD initiatives
| Test time bomb coupons |
|
|
CRM Impact | We have 172K users (46% of our email contactable base) who did not open an email in the last 180 days
| Test a more aggressive copy to reactivate lapsed customers (test different copy variants) |
|
|
Loyalty Sales | BKNL is behind average in terms of loyalty sales compared to EMEA main markets
Power BI - September YTD | Test refer-a-friend mechanics to leverage network effect to create incentives to join the program |
|
|
Loyalty Sales | Contribute to increase average check through second ticket
| Test dessert coupon push, 5, 10, and 15 minutes post transaction
|
|
|
Results
Burger King Germany
Opportunities
Increase contactable sales
Increase the loyalty base
Increase CRM Impact
A/B tests plan
Objective | Status | Test | Plan | Results Con |
|---|---|---|---|---|
Contactable Sales | Only 23% of the loyalty base is opted-in for communications |
|
|
|
Contactable Sales | Only 60% of the contactable base is actually being reached with communications |
|
|
|
Contactable Sales | 36% of the new sign ups are opted in
| Test different copy to incentivize opt in (e.g. want to unlock more coupons?) |
|
|
CRM Impact |
|
|
|
|
CRM Impact | We don’t have a ready to activate journey to react to McD initiatives
| Test time bomb coupons |
|
|
CRM Impact | We have 390K users (34% of our email contactable base) who did not open an email in the last 180 days https://app.amplitude.com/audiences/burgerkingde/cohort/r3qmaval?source=copy+url | Test a more aggressive copy to reactivate lapsed customers (test different copy variants) |
|
|
Loyalty Sales | Contribute to increase average check through second ticket
| Test dessert coupon push, 5, 10, and 15 minutes post transaction
|
|
|