Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 20 Next »

Documentation for CRM A/B Tests Catalog:

  • For each A/B test

    • Test category

    • Description of the test

    • Test performance per market (and month when test ended)

    • Visuals (if relevant)

  • This catalog is a non-exhaustive list of features/contents to A/B test and is recurrently being updated every time new relevant tests are done

  • The catalog only includes markets that have RBI recommended CRM Stack (Amplitude, Braze, mParticle) and focuses one the one serviced by RBI CRM services

🧐 KPIs

Results are measured based on start session and conversion uplift. The uplift is defined as follows:

  • % of recognized transactions test/ % of recognized transactions control

  • 7 days window after entering the specific canvas

βš™οΈ A/B Tests Catalogue

0️⃣1️⃣ Naming Personalization

  • Channel: push notifications

  • Description:

    • Version A: Special food is waiting for you! πŸ˜‹ Beat the Monday blues with a hearty Burger King meal πŸ”

    • Version B: Hey {{${first_name}}}, special food is waiting for you! πŸ˜‹ Beat the Monday blues with a hearty Burger King meal πŸ”

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023): +28% CVR purchase no personalization (version A winning, available here)

0️⃣2️⃣ Loyalty Reference

  • Channel: push notifications

  • Description:

    • Version A: Explore Tuesday's BK Classics! πŸ”πŸŸOrder today and unlock new layers of deliciousness πŸ‘‘

    • Version B: Explore Tuesday's BK Classics! πŸ”πŸŸOrder now and get crowns for every CHF you spend πŸ‘‘

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023): +12% CVR purchase with loyalty reference (version B winning), but both worse than control group

0️⃣3️⃣ Direct CTA

  • Channel: push notifications

  • Description:

    • Version A: Lazy Sunday Calls for BK Delights! 🍟 Lazy Sunday Sorted - BK Delivers Happiness! 🏠🍟 For T&C visit:

    • Version B: Lazy Sunday Calls for BK Delights! 🍟 Order now & enjoy BK's flavorful burgers! 🏠

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023): +4.5% CVR purchase without direct CTA (version A winning)

0️⃣4️⃣ Naming Personalization Placement

  • Channel: push notifications

  • Description:

    • Version A: name placement in the PH

    • Version B: name placement in the SL

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BK DE (Nov 2023): +25% CVR purchase with personalization in the SL versus pre-header (version B winning)

0️⃣5️⃣ Branded Copy

  • Channel: push notifications

  • Description:

    • Version A: πŸ”πŸ‘‘ Dienst am Dienstag? Wir auch!πŸ‘‘πŸ” Lass dich von uns versorgen! πŸ‘‘

    • Version B: πŸ”πŸ‘‘ Dienst am Dienstag? Wir auch!πŸ‘‘πŸ”πŸ‘‰ Lass deine Küche kalt und genieße einen flamed grilled BurgerπŸ”₯

  • Test significance: πŸ”΄

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BK DE (Nov 2023): no significant difference among the 2 variations.

0️⃣6️⃣ Abbreviations

  • Channel: push notifications

  • Description:

    • Version A: TGIF: Hochgenuss von Burger King®! πŸŽ‰

    • Version B: Freitag ist King πŸ‘‘

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BK DE (Nov 2023): +16% CVR purchase with young abbreviation (version A winning)

0️⃣7️⃣ Timing HD

Impact on orders +120% versus previous 4 Wednesdays (include vs LAST YEAR)

Date

2023-11-29

2023-12-06

2023-12-13

2023-12-20

2023-12-27

Orders

17

12

11

18

32

0️⃣8️⃣ Communication pressure

  • Channel: push notifications

  • Description:

    • Version A: receives all push notifications (including journeys & randomizer)

    • Version B: receives only journeys & randomizer, no marketing push

  • Test significance:

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023):

0️⃣9️⃣ App Download banner

  • Channel: Branch Journeys

  • Description:

    • Version A: upper banner

    • Version B: Bottom banner

image-20231221-151136.png

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡¨πŸ‡­ BK CH (Nov 2023): +47% conversion (to download), +18% conversion (to open)

image-20231221-151000.png

Uplift bottom banner only:

image-20240103-115445.png

1️⃣0️⃣ Direct CTA in the Subject Line

  • Channel: email

  • Description: different pre-header and subject line. One is more cheesy, the second one leverages urgency to boost visit

    • Version A: Subject: Große Freude, kleiner Preis! 🍟 Preheader: 🚨 Zu jedem King Menü ein King Jr.® Meal für nur 1€!

    • Version B: Subject: Hol es dir 🚨 Preheader: Zu jedem King Menü ein King Jr.® Meal für nur 1€!

  • Test significance: 🟒

  • Test performance per market:

    • πŸ‡©πŸ‡ͺ BKDE (Dec’23) Version B has higher open rate (+0.6%), higher CTO (+1%) and lower un-subscription rate

image-20240104-103328.png
  • Nevertheless, version B has a lower conversion rate than version A (-3%). Both version A and version B are higher than control

image-20240104-104343.png

image-20240104-103805.pngimage-20240104-103928.png

image-20240104-104032.png

1️⃣ 1️⃣ Segmentation

  • Channel: email

  • Description: different email content based on PMIX preference. Specifically, split between vegetarian & others.

  • Test significance: πŸ”΄ the test set up was incorrect, as we didn’t have a control group specifically among the vegetarian that received the “all” variants.

  • Test performance per market:

1️⃣ 1️⃣ IAM

  • Channel: in app messages

  • Description:

  • Test significance: πŸ”΄ the test set up was incorrect, as we didn’t have a control group specifically among the vegetarian that received the “all” variants.

  • Test performance per market:

βš™οΈ Campaigns Catalogue

1️⃣ Fight Back

  • Channel: email

  • Description: McD has launched an aggressive coupon campaign related to the New Year. By leveraging an existing template in the library, we have reacted in less than 24 hours pushing our coupons that are more aggressive

    • Anna, anno nuovo, nuove offerte!

image-20240104-162841.png
  • PH: New Year, More Savings! SL: Celebrate the new year with these special coupons for you ‌ ‌ ‌ ‌ ‌

image-20240104-162713.png

  • Test significance:

  • Test performance per market: πŸ‡¨πŸ‡­

  • No labels