3 tests for email campaigns: Part One – Emojis

We are big fans of new creative ideas and theory, but on the other hand theory is still only theory until proven. That is why we decided to run three A/B tests on new adoptions in email marketing utilising emojis in subject lines, gifs and content type. We didn’t want to attempt to discuss all of them in one post so this will be a three part series. First up the vogue subject of emojis.
If you have read any of our other blog posts on emojis (part one and part two) you will have seen that 2015 has been a big year for our funny little digital friends. Not only are Facebook rolling out their Emotions function but in the last week the ‘tears of joy face’ emoji has been named as the Oxford Dictionary’s word of the year. They have been described as a more concise and more personable way of conveying a message or emotion. We are inclined to agree – but are they suitable for email marketing? Experian’s research showed that over 50% experienced an increase in open rates with the use of emojis in their subject lines which sounded very promising. We put it to the test by sending out three e-shots with and without emojis in the subject line.

How we set it up the e-shot test

We can choose a custom list of contacts to which we would send out these tests. Our database is relatively small so we decided to use everyone on our contact list to see if we could get an overall view of the reaction to emojis. We were then able to choose the amount of people we would send the A/B test to and the length of time the test would run until the the type with the highest open rate would then be sent out to the rest of our contacts. We sided with 50% of our contacts at random for the A/B test which equated to 1500 contacts per email type. A further 3000 would be sent out to the other 50% using the most successful subject line.

Caveat: this was by no means a perfect test. In an ideal world we would send out these tests to a number of demographics to further segment the data from the e-shot and remove as many variables as possible. Regardless, it showed some interesting results.

Spoiler alert: if you don’t want to read the statistics here is a little summary. Overall the emoji subject did not fare well. Not badly, but in terms of open rate success, it lagged over 2% on average less than the subject line without emojis. For a debrief on why this might have happened scroll down past the test stats!

The stats

Test 1

Sent on a Wednesday afternoon at 14:35 we had one subject line with no emojis and one with two emojis (at the beginning and at the end). Within 15 minutes the subject line without emojis had gained a 2.5% lead in open rates, stretching this out to over 4% in the subsequent 2 hours and 10 minutes. For all of the below graphs the blue line represents the emoji subject line and the red line represents the text based subject line.

Test 2The second test was somewhat hampered by a technical issues which meant that over 65% of the emails were not delivered. With our email campaign schedule dictated by our own content creation strategy we therefore had use this incomplete send as test 2 and the remainder as test 3. For test 2 we decided to use the same format with the emoji subject line with one emoji at the beginning and one at the end. We also decided to send it mid morning in order to see if our dreary eyed recipients would have their days lightened by an amusingly emojified subject line! Different emojis were used this time for the two emojis and initially this seemed to have a more positive impact with the emoji subject line holding a 1-1.5% increase in open rate for the first 40 minutes or so. The end result though was a close run contest that resulted in a draw between the two subject lines after a longer test period of over 4 hours.Test 3Using the contacts that had failed to receive the emails from the second test we switched up the format of the emoji subject line only using one emoji at the beginning. Overall the email with the emoji subject line lagged behind by around 2% open rate for the majority of the 2 hour and 20 minute test, only closing the gap to within 0.5% in the final 15 minutes.

The results

As already stated above this was no perfect test. However, the overall stats point towards emojis, in these instances, not being as successful in terms of open rates. Now this does not mean that Experian were wrong – it just means that emojis were not suitable for these 3 tests. Marketing is not a one dimensional practice and this has been just another reminder of that fact. There are multiple factors that may have contributed towards a less successful emoji based subject line than was originally expected:– A lot of our database is made up of business owners and marketing directors rather than Generation Y and Z who are more suited to the use of emojis. As such, as a B2B service rather than a B2C service our target demographic are perhaps less likely to be attracted by emojis within subject lines.– Email filters and compatibility will no doubt have affected the ability of our emoji subject line email to find its way into the inbox of our targets. When we tested internally, sending the email to our own personal accounts, the emoji subject line was unanimously placed in the promotions section of gmail rather than the primary inbox. With the widespread use of gmail for both personal and business email accounts this would dramatically reduce the emoji subject line’s effectiveness. Furthermore, as discussed in our previous post on emojis, some iterations of Outlook are unlikely to display the emoji at all. Instead the recipient will be displayed a nondescript box or even the word emoji as a replacement. In fact, we have seen gmail experience issues displaying some emojis.Hardly the desired impact!It could be argued that the emoji itself was simply not eye catching enough or were not relevant enough to the subject line text. However, we would counter this by the fact that we used 3 different emojis and subject lines with two different variations of delivery which helps remove these variables. Furthermore, the emoji subject line had an 8.5% increase in open rates compared with previous e-shots despite still losing out in open rates to the simple text based subject line.Again, as is the case with all marketing, relevancy is a key metric. For example, Experian commented that the umbrella emoji had one of the highest open rates but did not state whether this had been sent on a rainy day or not. We will look to utilise emojis that are relevant to more factors than just the content of the email in order to test whether this has an effect on open rates and subsequent engagement. We will continue to run extended tests on this using more segmented data and report back! Next stage is to run a test for longer considering that test 3 showed a fightback from the emoji subject line in the last 15 minutes of the test, leaving us wondering what would have happened if we had left it for longer…Next up – using gifs in e-shots.