Documentation for CRM A/B Tests Catalog:
For each A/B test
Test category
Description of the test
Test performance per market (and month when test ended)
Visuals (if relevant)
This catalog is a non-exhaustive list of features/contents to A/B test and is recurrently being updated every time new relevant tests are done
The catalog only includes markets that have RBI recommended CRM Stack (Amplitude, Braze, mParticle) and focuses one the one serviced by RBI CRM services
🧐 KPIs
Results are measured based on start session and conversion uplift. The uplift is defined as follows:
% of recognized transactions test/ % of recognized transactions control
7 days window after entering the specific canvas
⚙️ A/B Tests Catalogue
0️⃣1️⃣ Naming Personalization
Channel: push notifications
Description:
Version A: Special food is waiting for you! 😋 Beat the Monday blues with a hearty Burger King meal 🍔
Version B: Hey {{${first_name}}}, special food is waiting for you! 😋 Beat the Monday blues with a hearty Burger King meal 🍔
Test significance: 🟢
Test performance per market:
🇨🇭 BK CH (Nov 2023): +28% CVR purchase no personalization (version A winning, available here)
0️⃣2️⃣ Loyalty Reference
Channel: push notifications
Description:
Version A: Explore Tuesday's BK Classics! 🍔🍟Order today and unlock new layers of deliciousness 👑
Version B: Explore Tuesday's BK Classics! 🍔🍟Order now and get crowns for every CHF you spend 👑
Test significance: 🟢
Test performance per market:
🇨🇭 BK CH (Nov 2023): +12% CVR purchase with loyalty reference (version B winning), but both worse than control group
0️⃣3️⃣ Direct CTA
Channel: push notifications
Description:
Version A: Lazy Sunday Calls for BK Delights! 🍟 Lazy Sunday Sorted - BK Delivers Happiness! 🏠🍟 For T&C visit:
Version B: Lazy Sunday Calls for BK Delights! 🍟 Order now & enjoy BK's flavorful burgers! 🏠
Test significance: 🟢
Test performance per market:
🇨🇭 BK CH (Nov 2023): +4.5% CVR purchase without direct CTA (version A winning)
0️⃣4️⃣ Naming Personalization Placement
Channel: push notifications
Description:
Version A: name placement in the PH
Version B: name placement in the SL
Test significance: 🟢
Test performance per market:
🇩🇪 BK DE (Nov 2023): +25% CVR purchase with personalization in the SL versus pre-header (version B winning)
0️⃣5️⃣ Branded Copy
Channel: push notifications
Description:
Version A: 🍔👑 Dienst am Dienstag? Wir auch!👑🍔 Lass dich von uns versorgen! 👑
Version B: 🍔👑 Dienst am Dienstag? Wir auch!👑🍔👉 Lass deine Küche kalt und genieße einen flamed grilled Burger🔥
Test significance: 🔴
Test performance per market:
🇩🇪 BK DE (Nov 2023): no significant difference among the 2 variations.
0️⃣6️⃣ Abbreviations
Channel: push notifications
Description:
Version A: TGIF: Hochgenuss von Burger King®! 🎉
Version B: Freitag ist King 👑
Test significance: 🟢
Test performance per market:
🇩🇪 BK DE (Nov 2023): +16% CVR purchase with young abbreviation (version A winning)
0️⃣7️⃣ Timing HD
Channel: push notifications
Description:
Version A: 11.45 am
Version B: 5.45pm
Test significance: 🔴 only 17 conversion in version B
Test performance per market:
Version B conversion rate: 6.5%
Version A conversion rate: 10.5%
Control group conversion rate is 4.8%
Impact on orders +120% versus previous 4 Wednesdays (include vs LAST YEAR)
Date | 2023-11-29 | 2023-12-06 | 2023-12-13 | 2023-12-20 | 2023-12-27 |
Orders | 17 | 12 | 11 | 18 | 32 |
0️⃣8️⃣ Communication pressure
Channel: push notifications
Description:
Version A: receives all push notifications (including journeys & randomizer)
Version B: receives only journeys & randomizer, no marketing push
Test significance:
Test performance per market:
🇨🇭 BK CH (Nov 2023):
0️⃣9️⃣ App Download banner
Channel: Branch Journeys
Description:
Version A: upper banner
Version B: Bottom banner
Test significance: 🟢
Test performance per market:
🇨🇭 BK CH (Nov 2023): +47% conversion (to download), +18% conversion (to open)
Uplift bottom banner only:
1️⃣0️⃣ Direct CTA in the Subject Line
Channel: email
Description: different pre-header and subject line. One is more cheesy, the second one leverages urgency to boost visit
Version A: Subject: Große Freude, kleiner Preis! 🍟 Preheader: 🚨 Zu jedem King Menü ein King Jr.® Meal für nur 1€!
Version B: Subject: Hol es dir 🚨 Preheader: Zu jedem King Menü ein King Jr.® Meal für nur 1€!
Test significance: 🟢
Test performance per market:
🇩🇪 BKDE (Dec’23) Version B has higher open rate (+0.6%), higher CTO (+1%) and lower un-subscription rate
Nevertheless, version B has a lower conversion rate than version A (-3%). Both version A and version B are higher than control
1️⃣ 1️⃣ Segmentation
Channel: email
Description: different email content based on PMIX preference. Specifically, split between vegetarian & others.
Test significance: 🔴 the test set up was incorrect, as we didn’t have a control group specifically among the vegetarian that received the “all” variants.
Test performance per market:
1️⃣2️⃣ IAM Impact on User Behavior
Channel: in app messages
Description: for the Veganuary campaign, on 12/01 we will activate IAM and we aim to check the impact of these on the guest behavior
Test significance:
Test performance per market: 🇩🇪 BKDE (Jan’24)
1️⃣3️⃣ Communication Frequency
Channel: push notifications
Description: increased communication pression during the holiday period (24/12-07/01) resulting in ~10 push notifications in the period
Test significance:
Test performance per market: 🇩🇪 BKDE (Dec’23)
the test grow vs control growth was ~11% (versus the 3.5% average of the previous weeks). Therefore, it seems that the increased communication pressure has influenced positively consumer behavior.
Week | 2023-12-11 | 2023-12-18 | 2023-12-25 | 2024-1-1 |
Sales Test Growth – Sales Control Growth | 2.64% | 4.74% | 11.39% | 10.60% |
The Amplitude chart can be found here: https://app.amplitude.com/analytics/burgerkingde/chart/o4yf9tm9?linkingDashboardId=482el67g
Version A:
Subject: Überraschung zum Veganuary! 🌱🍔
Preheader: 🆕 Schon den neuen Veggie King Deluxe ausgecheckt?
Version B:
Subject: So lecker isst 2024! 🌱🍔
Preheader: 🆕 Schon den neuen Veggie King Deluxe ausgecheckt?
1️⃣4️⃣ Segments & Protein Preference
Channel: email & push notification
Description: we have sent an email for Veganuary, and we have checked the performances of the segments based on their protein preferences
Test significance: 🟡 (we don’t have yet the full conversion window)
Test performance per market:
Vegetarian have better conversion rate, followed by chicken and beef
We see a resistance for beef users to purchase vegeterian products
Insights:
We test grouping chicken & veggie together when we have segmented content based on protein
We can test different subject line for beef users
1️⃣5️⃣ Raffle
Test performance per market:
The easiest the mechanic is, the more likely are guests to engage with the raffle
Engaging content for CRM: Raffle have above average performances for open rate and CTO, with low un-subscription
Engaging content for the App: Users who participate in the raffle have better activity rate than non-participants
Insights:
A/B test PH/SL needs to be more explicit (win, raffle, opt in).
Push
1️⃣6️⃣ Open Rate
Channel: email & push notification
Description: we have sent an email for Veganuary, and we have checked the performances of the segments based on their protein preferences
Test significance: 🟡 (we don’t have yet the full conversion window)
⚙️ Campaigns Catalogue
1️⃣ Fight Back
Channel: email & push notifiactions
Description: McD has launched an aggressive coupon campaign related to the New Year. By leveraging an existing template in the library, we have reacted in less than 24 hours pushing our coupons that are more aggressive through email. After that, we have introduced 3 push notifications (10/1, 12/1, 14/1)
Anna, anno nuovo, nuove offerte!
PH: New Year, More Savings! SL: Celebrate the new year with these special coupons for you
Test significance:
Test performance per market: 🇨🇭