Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 33 Next »

Documentation for CRM A/B Tests Catalog:

  • For each A/B test

    • Test category

    • Description of the test

    • Test performance per market (and month when test ended)

    • Visuals (if relevant)

  • This catalog is a non-exhaustive list of features/contents to A/B test and is recurrently being updated every time new relevant tests are done

  • The catalog only includes markets that have RBI recommended CRM Stack (Amplitude, Braze, mParticle) and focuses one the one serviced by RBI CRM services

🧐 KPIs

Results are measured based on start session and conversion uplift. The uplift is defined as follows:

  • % of recognized transactions test/ % of recognized transactions control

  • 7 days window after entering the specific canvas

βš™οΈ A/B Tests Catalogue

0️⃣1️⃣ Naming Personalization

  • Channel: push notifications

  • Description:

    • Version A: Special food is waiting for you! πŸ˜‹ Beat the Monday blues with a hearty Burger King meal πŸ”

    • Version B: Hey {{${first_name}}}, special food is waiting for you! πŸ˜‹ Beat the Monday blues with a hearty Burger King meal πŸ”

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023): +28% CVR purchase no personalization (version A winning, available here)

  • McD CH is leveraging first name and emoj in all communications

image-20240213-125254.png

0️⃣2️⃣ Loyalty Reference

  • Channel: push notifications

  • Description:

    • Version A: Explore Tuesday's BK Classics! πŸ”πŸŸOrder today and unlock new layers of deliciousness πŸ‘‘

    • Version B: Explore Tuesday's BK Classics! πŸ”πŸŸOrder now and get crowns for every CHF you spend πŸ‘‘

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023): +12% CVR purchase with loyalty reference (version B winning), but both worse than control group

0️⃣3️⃣ Direct CTA

  • Channel: push notifications

  • Description:

    • Version A: Lazy Sunday Calls for BK Delights! 🍟 Lazy Sunday Sorted - BK Delivers Happiness! 🏠🍟 For T&C visit:

    • Version B: Lazy Sunday Calls for BK Delights! 🍟 Order now & enjoy BK's flavorful burgers! 🏠

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023): +4.5% CVR purchase without direct CTA (version A winning)

0️⃣4️⃣ Naming Personalization Placement

  • Channel: push notifications

  • Description:

    • Version A: name placement in the PH

    • Version B: name placement in the SL

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BK DE (Nov 2023): +25% CVR purchase with personalization in the SL versus pre-header (version B winning)

  • McD Germany is leveraging emoji at the beginning of each communications

image-20240213-125148.png

0️⃣5️⃣ Branded Copy

  • Channel: push notifications

  • Description:

    • Version A: πŸ”πŸ‘‘ Dienst am Dienstag? Wir auch!πŸ‘‘πŸ” Lass dich von uns versorgen! πŸ‘‘

    • Version B: πŸ”πŸ‘‘ Dienst am Dienstag? Wir auch!πŸ‘‘πŸ”πŸ‘‰ Lass deine Küche kalt und genieße einen flamed grilled BurgerπŸ”₯

  • Test significance: πŸ”΄

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BK DE (Nov 2023): no significant difference among the 2 variations.

0️⃣6️⃣ Abbreviations

  • Channel: push notifications

  • Description:

    • Version A: TGIF: Hochgenuss von Burger King®! πŸŽ‰

    • Version B: Freitag ist King πŸ‘‘

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BK DE (Nov 2023): +16% CVR purchase with young abbreviation (version A winning)

0️⃣7️⃣ Timing HD

Impact on orders +120% versus previous 4 Wednesdays (include vs LAST YEAR)

Date

2023-11-29

2023-12-06

2023-12-13

2023-12-20

2023-12-27

Orders

17

12

11

18

32

0️⃣8️⃣ Communication pressure

  • Channel: push notifications

  • Description:

    • Version A: receives all push notifications (including journeys & randomizer)

    • Version B: receives only journeys & randomizer, no marketing push

  • Test significance:

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023):

0️⃣9️⃣ App Download banner

  • Channel: Branch Journeys

  • Description:

    • Version A: upper banner

    • Version B: Bottom banner

image-20231221-151136.png

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023 - IOS only): +47% conversion (to download), +18% conversion (to open)

image-20231221-151000.png

Uplift bottom banner only:

image-20240103-115445.png

Similar results for Android banner:

image-20240131-082451.png

1️⃣0️⃣ Direct CTA in the Subject Line

  • Channel: email

  • Description: different pre-header and subject line. One is more cheesy, the second one leverages urgency to boost visit

    • Version A: Subject: Große Freude, kleiner Preis! 🍟 Preheader: 🚨 Zu jedem King Menü ein King Jr.® Meal für nur 1€!

    • Version B: Subject: Hol es dir 🚨 Preheader: Zu jedem King Menü ein King Jr.® Meal für nur 1€!

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BKDE (Dec’23) Version B has higher open rate (+0.6%), higher CTO (+1%) and lower un-subscription rate

image-20240104-103328.png
  • Nevertheless, version B has a lower conversion rate than version A (-3%). Both version A and version B are higher than control

image-20240104-104343.png

image-20240104-103805.pngimage-20240104-103928.png

image-20240104-104032.png

1️⃣ 1️⃣ Segmentation

  • Channel: email

  • Description: different email content based on PMIX preference. Specifically, split between vegetarian & others.

  • Test significance: πŸ”΄ the test set up was incorrect, as we didn’t have a control group specifically among the vegetarian that received the “all” variants.

  • Test performance per market:

1️⃣2️⃣ IAM Impact on User Behavior

  • Channel: in app messages

  • Description: for the Veganuary campaign, on 12/01 we will activate IAM and we aim to check the impact of these on the guest behavior

  • Test significance:

  • Test performance per market: πŸ‡©πŸ‡ͺ BKDE (Jan’24)

1️⃣3️⃣ Communication Frequency

  • Channel: push notifications

  • Description: increased communication pression during the holiday period (24/12-07/01) resulting in ~10 push notifications in the period

  • Test significance:

  • Test performance per market: πŸ‡©πŸ‡ͺ BKDE (Dec’23)

  • the test  grow vs control growth was ~11% (versus the 3.5% average of the previous weeks). Therefore, it seems that the increased communication pressure has influenced positively consumer behavior.

 

Week

2023-12-11

2023-12-18

2023-12-25

2024-1-1

Sales Test Growth – Sales Control Growth

2.64%

4.74%

11.39%

10.60%

 

The Amplitude chart can be found here: https://app.amplitude.com/analytics/burgerkingde/chart/o4yf9tm9?linkingDashboardId=482el67g

image-20240109-221510.png

1️⃣4️⃣ Segments & Protein Preference

  • Channel: email & push notification

  • Description: we have sent an email for Veganuary, and we have checked the performances of the segments based on their protein preferences

  • Test significance: 🟑 (we don’t have yet the full conversion window)

  • Test performance per market:

    • Vegetarian have better conversion rate, followed by chicken and beef

    • We see a resistance for beef users to purchase vegeterian products

  • Insights:

    • We test grouping chicken & veggie together when we have segmented content based on protein

    • We can test different subject line for beef users

image-20240116-102750.png

Push

image-20240116-102945.png

1️⃣5️⃣ PH & SL

  • Channel: email & push notification

  • Description: test short vs long version for the PH & SL

Version A:

  • Subject: Überraschung zum Veganuary! πŸŒ±πŸ”

  • Preheader: πŸ†• Schon den neuen Veggie King Deluxe ausgecheckt?

Version B:

  • Subject: So lecker isst 2024! πŸŒ±πŸ”

  • Preheader: πŸ†• Schon den neuen Veggie King Deluxe ausgecheckt?

  • Test significance: 🟑 (we don’t have yet the full conversion window)

  • Test performance per market:

    • For push notifications, version A worked better

    • For email, version B worked better

1️⃣6️⃣ Open Rate

  • Channel: email

  • Description: we have sent the communication at 11.45 and we wanted to check if the users are opening it during that time or if there are other opening hours we should consider

  • Test significance: -

  • Test performance per market: πŸ‡©πŸ‡ͺ BKDE (Q4 2023)

    • People open the email immediately after receiving it (spike between 11-12)

    • There is a also some activity the day after, at 8am

    • Next: A/B test 10.45 vs 11.45

image-20240116-104459.png

1️⃣7️⃣ Pricing vs FREE

  • Channel: push notifications

  • Description:

Communicating Long Chicken Medium Menu with FREE 6 Chili Cheese Nuggets
Communicating Long Chicken Medium Menu + 6 Chili Cheese Nuggets for 14.40

  • Test significance: 🟒

  • Test performance per market: πŸ‡¨πŸ‡­ BKCH (Q1 2024)

    • General copy outperforms both for start session and for conversion

image-20240304-095914.png

1️⃣7️⃣ Urgency

  • Channel: push notifications

  • Description:

A/B test on urgency

  • Last 3 days to get your Big King Medium Menu for 10.90
    Get your Big King Medium Menu for 10.90

  • Test significance: 🟒

  • Test performance per market: πŸ‡¨πŸ‡­ BKCH (Q1 2024)

    • Urgency copy outperforms for both start session & conversion rate

image-20240304-103838.png

1️⃣8️⃣ Send Time

  • Channel: push notifications

  • Description: best send time for push notifications sends during the week (Wednesday). A/B testing 3pm vs 6pm

  • Test significance: 🟒

  • Test performance per market: πŸ‡³πŸ‡± BKNL (Q1 2024)

    • 6pm overperforms 3pm both in open rate and conversion rate

    • Next Steps:

      • Lunch vs Dinner

      • Winning option vs intelligent timing

image-20240304-111238.png

1️⃣9️⃣ PMIX - Declared vs Actual

  • Channel: push notifications

  • Description: analyzing the offer redemption based on the PMIX preferences declared in the welcome survey

    • Crispy Chicken added to offer more for both the chicken offer receivers & beef offer receivers,

    • Highest conversion is chicken lovers <> purchase chicken

    • For Veggie: Highest conversion actually for chicken <> beef than for veggie lovers

  • Test significance: 🟒

  • Test performance per market: πŸ‡¨πŸ‡­ BKCH (Q1 2024)

    • Next steps: send out segmented push notification based on transaction (vs send based on welcome survey answers)

    • It seems that people are declaring to be vegetarian but keep purchasing other products

image-20240304-105736.png

2️⃣ 0️⃣ Loyalty Points as CTA

  • Channel: push notifications

  • Description: analyzing which products and related points to redeem are more engaging for the guest. We have tested 700 (Free Whopper) and 1000 (Double Whopper). The hypothesis is the option with the lower point balance is the best as it seems more achievable

  • Test significance: 🟒

  • Test performance per market: πŸ‡³πŸ‡± BKNL (Q1 2024)

    • 700 outperformed 1000

    • Next steps:

      • Repeat the test and switch order. Currently we have advertised product = crowns, we could try crowns = product

      • Another test could be to identify if the performances were impacted by the choice of the product (Double Whopper less attractive than Whopper). Therefore we can try to substitute Double Whopper with LTO

image-20240312-091503.png

βš™οΈ Campaigns Catalogue

1️⃣ Fight Back

  • Channel: email & push notifiactions

  • Description: McD has launched an aggressive coupon campaign related to the New Year. By leveraging an existing template in the library, we have reacted in less than 24 hours pushing our coupons that are more aggressive through email. After that, we have introduced 3 push notifications (10/1, 12/1, 14/1)

    • Anna, anno nuovo, nuove offerte!

image-20240104-162841.png
  • PH: New Year, More Savings! SL: Celebrate the new year with these special coupons for you ‌ ‌ ‌ ‌ ‌

image-20240104-162713.png

  • Test significance:

  • Test performance per market: πŸ‡¨πŸ‡­

2️⃣ Raffle

  • Test performance per market: πŸ‡©πŸ‡ͺ BKDE (Q4 2023)

    • The easiest the mechanic is, the more likely are guests to engage with the raffle

image-20240116-111750.png
  • Push notifications had higher conversion to opt in compared to in app messages

image-20240116-111854.png

  • Engaging content for CRM: Raffle have above average performances for open rate and CTO, with low un-subscription

image-20240116-111935.png

  • Engaging content for the App: Users who participate in the raffle have better activity rate and conversion rate than non-participants

image-20240116-111916.png

See here full dashboard with insights: https://app.amplitude.com/analytics/burgerkingde/dashboard/sgfef3pk

  • Insights:

    • A/B test PH/SL needs to be more explicit (win, raffle, opt in).

  • No labels