##A/B testing the effectiveness of HTML vs. AMP emails

AMP for emails is a relatively new technology, and many marketers are still unsure whether it's worth using, as AMP emails require more time to create and therefore have a higher cost. In collaboration with Mail.ru Cloud, we prepared an AMP campaign targeting inactive Cloud users. The goal was not only to reactivate them but also to verify the effectiveness of using AMP for this purpose. The email was sent as an A/B test: one version was a standard HTML email offering users an additional 256GB for 2 months, while the second version was a full-fledged AMP email with a 'find the object' game mechanic.

<Img
img="/api/uploads/blog_2_1_e749851faa.gif"
alt="Myths About Marketing Automation"
/>
Both versions were sent with the same subject line and had approximately equal open rates. 

|    | HTML   | AMP    | 
|----|--------|--------|
| OR | 11.11% | 10.19% |

It should be noted that the emails were sent to inactive users. 

Not all users who received the AMP version were able to view it due to limited support (e.g., they opened the email in a mail client that does not support AMP).

AMP
|AMP-part|	HTML-part|
|---------------|-----------------|
|81%           |19%|

The proportions differ from the typical 50/50 split because this campaign was specifically targeted at Mail.ru users, where AMP is supported in both mobile apps and the web interface.

While the HTML version (in the A/B test) simply required users to click a link to claim their reward, the AMP version involved a gamified experience where users had to find hidden objects before they could access the reward. Despite the added step, we observed a 78% increase in clicks to the website from the AMP version.

|HTML|	AMP|
|----------------|----------------|
|CTOR|	4.88%|	8.67%|

(Clicks to the web version of the email were not included in the statistics.) 

In other words, engaging the user with a game increased clicks from the email by 78%. Once a subscriber took the initial action within the email, it became more difficult for them to stop. The primary goal of the email was to activate the gift, and we saw a 55% increase in gift activations from the AMP email compared to the HTML email. In addition to standard email metrics, we also measured the average time spent in the email, which was 10.8 seconds. Notably, users who spent more than 10 seconds in the email completed the game and clicked the link.

##Results

Implementing AMP gamification in the email resulted in a 78% increase in click-through rate (CTOR) and a 55% increase in conversion rate. A crucial aspect is the psychological factor: to get a user to perform a desired action, it's necessary to hold their attention for more than 10 seconds.


We implemented AMP-powered gamification in emails and conducted A/B testing

small_article3.jpg

thumbnail_article3.jpg

article3.jpg

HTML vs. AMP: A 55% Conversion Boost

##First pancake

Together with Math & Mind, we developed an AMP email, where we asked the user to answer several questions directly in the email.

The original version of the letter looked like this:

<Img
img="/api/uploads/blog1_2aaec46973.png"
alt="Myths About Marketing Automation"
/>
We sent it to a small test sample to check how this mechanic would work. And, to our surprise, the letter did not show any results different from a regular html letter.

##Work on mistakes

The fact is that subscribers are not yet accustomed to interactive letters, especially since it was visually no different from what was sent before. Users expected that this was the most ordinary letter and that upon clicking on the answer they would be redirected to the site.

To visually “revive” the letter and show that you can interact with it, we added a carousel on top with an increasing timer “your brain age”

<Img
img="/api/uploads/blog2_b3f51b0b84.png"
alt="Myths About Marketing Automation"
/>

This turned out to be the missing link: the test sample showed good results, and the letter was sent to the main audience.

##Results

We tracked the openings of the HTML and AMP versions separately. Unique discoveries were distributed as follows:

*HTML – 64.63%
*AMP – 35.37%
If you look at non-unique discoveries, then:

*HTML – 62.8%
*AMP – 37.2%

<Img img="/api/uploads/blog3_5c9cc8d8b8.png" alt="Myths About Marketing Automation" />

This allows us to draw an unambiguous conclusion: people are more likely to return to AMP emails.

To find out how users behave inside the letter, we installed tracking pixels on each question in our quiz: to understand the outflow of the audience, that is, how interesting such mechanics are to subscribers.

<Img img="/api/uploads/blog4_a35eca50f2.png" alt="Myths About Marketing Automation" />

The first step is 100% of users who opened the AMP version of the letter.
Answered the first question – 27.9%
On the second – 25.7%
And for the last one – 24.8%.
We see minimal audience outflow. That is, if the user has taken the first step, then with a high probability he will go all the way.

After passing this mini-quiz, we invite the user to go to the site and play some more. And here users are more loyal and interested, so they willingly go to the site. We separately tracked transitions to the site from the HTML and AMP versions of the email and these are the values ​​we got:

HTML – 6.95%
AMP – 32.54%

<Img img="/api/uploads/blog5_3b20ad6835.png" alt="Myths About Marketing Automation" />

This is how you can graphically present the resulting OR and CTR indicators with highlighting what part is accounted for by HTML and AMP openings and transitions.

<Img img="/api/uploads/blog6_71fbed4eeb.png" alt="Myths About Marketing Automation" />

From this we can conclude that the use of AMP technology and the gradual involvement of the subscriber has significantly increased the CTR!

We implemented AMP technology and gradually engaged our subscribers.

small_article2.jpg

thumbnail_article2.jpg

article2.jpg

2.5x Increase in Click-Through Rate 

##Idea

Often, we need feedback from subscribers to understand their concerns, interests, and dislikes regarding our emails. Merely analyzing open rates (OR) and click-through rates (CTR) provides limited insights, prompting marketers to resort to subscriber surveys. However, low conversion rates can hinder survey completion, as many subscribers are reluctant to leave the email and answer questions.

To address this challenge, we leveraged AMP technology to embed interactive forms directly within the email. This eliminates the need for subscribers to navigate away from the email to participate.

To further motivate engagement, we gamified the survey by incorporating a magic ball that would answer the subscriber's questions. 

The user's question was saved on the server, and a random response ("yes," "no," or "I don't know") was displayed within the email. 

<Img
img="/api/uploads/blog_3_1_0d41492e6b.gif"
alt="Myths About Marketing Automation"
/>
###Metrics 

AMP doesn't work for everyone, so we collected and measured HTML and AMP version openings separately. 

Unique opens were distributed as follows:

*AMP – 56.3%
* HTML – 43.7%

If we look at non-unique opens, then:

*AMP – 62.13%
* HTML – 37.87%

<Img
img="/api/uploads/blog_3_2_2f4f2b48b6.png"
alt="Myths About Marketing Automation"
/>

When comparing the ratio of unique and non-unique opens between the HTML and AMP versions of the email, it becomes evident that users returned to AMP emails more frequently. 

On average, users returned to an AMP email 1.36 times, compared to 1.07 times for an HTML email.


###Form Submission Conversion

On average, for every user who opened the AMP version, there were **1.69** form submissions. This means that many subscribers filled out and submitted the form multiple times.

Even when considering only unique users who submitted the form, the result is still impressive: **20.74%.**

The game mechanics had an effect, and many of those who decided to submit the form did so several more times afterward. It was particularly interesting to observe how a subscriber would ask the same question to the "Magic Ball" until they received the desired answer.

We gamified the survey and saw these results.

small_article1.jpg

thumbnail_article1.jpg

article1.jpg

169% Conversion to Form Completion

Welcome emails are the first point of communication with a user, making it crucial to leave a good impression on subscribers. This task becomes significantly more challenging when the recipients are children.

While the main difficulty with adult audiences is getting them to open [**emails**](https://altcraft.com/blog/how-to-reach-customers-via-email-campaign) due to the overwhelming number of messages they receive, the challenge with emails for children is the lack of content that is suitable and engaging for them.

Therefore, for the Kids' Mail service, together with VK, we decided to create a welcome email sequence featuring content in a playful format. The main idea is to present the available services in such a way that children can build their ideal mail experience on their own.

## Concept Development

Just before we started working on the project, Kids' Mail updated its branding and introduced new mascots — cute little monsters. We decided to give users the chance to create their own "mail monster."

<Img
img="/api/uploads/pismo_struktura_285cf05547.png"
alt="Concept development"
/>

As we developed this concept, we also decided to introduce young subscribers to each of the services by giving a brief overview of its key features. When you click on a service, a description bubble appears.

<Img
img="/api/uploads/amp_email_build_your_monster_851afd1378.png"
alt="Email variant"
/>

At this stage, Mail's design team had already worked out the visual appearance and main features of our mail monster.
The monster turned out to be so adorable that we came up with another idea — to allow users to save the image they created and use it as, for example, a phone wallpaper.

## A Few Technical Details

Naturally, to implement such an interactive email, we could only use [**AMP**](https://pixcraft.io/blog/how-to-implement-amp-emails-into-your-email-strategy). Since the email is primarily targeted at mail.ru users, the majority will see the interactive version.

To ensure proper rendering and speed up the process, we used the [**TJML framework**](https://pixcraft.io/framework). Regarding the specifics of AMP interactivity, each service had its own state, which was set to either 1 (active) or 0 (inactive). Depending on the state, we would show or hide a particular layer (such as a tail, ears, eyes, etc.).

To track which services were most popular among children, when creating the monster, we also generated a form that collected the selected services and, upon clicking "done," sent the data to Pixcraft.

In the final step, we show a picture of the monster with its name. There’s a small technical detail worth mentioning here. The issue is that mail.ru doesn't work with default states, meaning that the following type of entry will be ignored:

```xml
<amp-state id="names">
  <script type="application/json">
	["Tsifrosh", "Servicio", "Filer", "Otpravita", "Megaguy", "Mailia", "Chernover"]
  </script>
</amp-state>
```
</br>
At the same time, this would have worked correctly in Gmail.

The universal solution in our case was to assign an array of possible names directly to the span, referring to a random element of the array:

```xml
<span [text]="['Tsifrosh', 'Servicio', 'Filer', 'Otpravita', 'Megaguy', 'Mailia', 'Chernover'][floor(random() * 7)]"></span>
``` 
</br>

Phone wallpapers were also tricky. With 7 available services, it was possible to create 128 different monster combinations. Manually generating all of these would have taken a lot of time, so we wrote a small program to generate all the possible variants automatically.

<Img
img="/api/uploads/amp_en_50cc990b0a.gif"
alt="AMP email"
/>

## Statistics

After launching the email, we collected and analyzed the statistics over a month. Since we are talking about [**AMP emails**](https://altcraft.com/ru/blog/chto-takoe-amp-pisma), the first thing to note is the percentage of AMP opens:

<Img
img="/api/uploads/welcome_pismo_statistika_1542f81de4.png"
alt="Statistics"
/>

For unique opens — 62% for AMP, 38% for HTML. For non-unique opens — 66% for AMP, 34% for HTML. **The conversion rate for submitting the form** (clicking the "done" button) was 19.9% of those who opened the email. This indicates that users returned to the AMP emails multiple times, reopening them to create different monsters, meaning the goal of capturing users' interest was definitely achieved.
We also analyzed which services for kids were the most popular and in-demand, and the results showed the following picture:

|    |         | 
|------------|-----------------|
| Cloud     | 44.29%          | 
| Cartoons     | 46.20%          | 
| Math     | 44.85%          | 
| Games     | 49.52%          | 
| Drawing Tool     | 50.43%          | 
| Marusya     | 44.60%          | 
| Emoji     | 49.92%          | 

However, it's likely that kids were choosing the features they found most visually appealing for their monster, rather than the services that genuinely interested them. It's no surprise that the "Drawing Tool" was added by over 50% of the subscribers, as it added a cute brush-tail. "Emoji," selected by nearly half, gave the monster adorable eyes, and "Games," which were close in popularity, added wings.

## Conclusion

This level of detail in the welcome emails allows us to build a trusting relationship with the audience from the very start. We paid great attention to small details, carefully crafted the content, and genuinely aimed to make the communication process enjoyable for children. This approach helped us achieve excellent results.
A significant part of this success was due to the PixCraft platform. It made it quick and easy to create high-quality email templates and supported the backend functionality of the AMP emails.

In this article, we will discuss how we developed the concept of a welcome letter for the Kids' Mail service.

Articles by tag: Cases