Travelplanet.pl: A/B tests and personalization that sold 38.8% more on mobile

The challenge

Traffic moved from desktop to mobile, but sales did not follow. People searched for trips on their phones and bought on a computer.

A trip is a big purchase, and most visitors left before the pre-order form. More than 84% of those who reached it abandoned it, and Travelplanet learned nothing about who they were or what they wanted.

A testing program, not one test

We have run more than 20 A/B tests and long-term analyses on the site. Each one starts with a hypothesis, runs for weeks and ends with a decision: implement, drop or test again. The ones that changed the business:

• The pre-order form moved onto the offer page, instead of a separate step. After more than 70 days of testing it brought 469 more bookings, and conversion kept rising in the months after it went live.
• A new offer page where users change dates and options without extra windows: conversion 0.26% → 0.29%.
• The price per person on the list buttons: conversion 0.32% → 0.34%.
• White tiles on the mobile category list: mobile conversion 0.08% → 0.11%.
• Travel insurance shown as a guarantee, not a warning: 0.40% against 0.31% for the version that mentioned a tour operator going bankrupt.

What did not work, and was dropped

A test that loses is still an answer, and it stops the team from shipping a change that would have cost sales. We explain how to run such tests in multivariate testing vs A/B testing.

• A sticky bar with a call to action on the offer page: no change after 22 days, removed.
• The “Good choice” filter switched on by default: a slightly higher basket, but fewer bookings.
• A trimmed-down mobile site: the same results without many boxes, which showed what could go next.

What we built

• A real-time personalization platform, added to the site with a single tag.
• Experiments, pop-ups, banners and push messages managed from one admin panel, without developers.
• Segmentation built on what each user viewed, and more than 20 panel modules, from lead handling and email to products and orders.
• Integrations with the MerlinX booking engine, the Travelplanet API, Google Analytics and Google Tag Manager, plus geolocation and SMS.
• Tracking sized for Travelplanet’s traffic: after 1.45 billion events the platform still ran at full speed, and a two-day load test with it switched on and off showed no effect on page speed.

Results

• 38.8% more sales on mobile than in the previous period, from personalizations and mobile UX changes, with Firecrux AIRE keeping the combination that sold best.
• 79% higher conversion from push notifications. One personalization shown to all traffic collected 88,707 opt-ins, and 41 messages to about 16,000 subscribers brought 22 bookings worth PLN 99,085.
• 2,291 callback leads in the first 18 months, from a pop-up that asks for a phone number when a user keeps looking at one destination.
• Product recommendations in the summer guide: transactions +44.15%, revenue +51.71% and e-commerce conversion +8.59%, year on year.

Checked again, against a control group

Before each new version of the site we check every personalization against a control group in Google Analytics and bring back only what works.

• The mini list on the offer page: e-commerce conversion 0.59% against 0.41% in the control group, 44% higher, over four months.
• Similar offers, reminder pop-ups and category blocks: users who clicked them converted 2 to 3 times as often as the control group. Part of that is self-selection, since clickers are keener buyers, so we read it as a signal, not as the size of the uplift.
• A “What you get” box and a price guarantee checkbox: no effect on conversion, so they did not come back.

What comes next

The work never had a finish line. Travelplanet brings us the next question when the offer, the site or the market changes, and we answer it the same way: a test first, then a decision.

• New personalizations and tests for each version of the site.
• Changes to the integrations as the booking engine and the offer change.