We are big fans of new creative ideas and theory, but on the other hand theory is still only theory until proven. That is why we decided to run three A/B tests on new adoptions in email marketing utilising emojis in subject lines, gifs and content type. We didn’t want to attempt to discuss all of them in one post so this will be a three part series. First up the vogue subject of emojis.
If you have read any of our other blog posts on emojis (part one and part two) you will have seen that 2015 has been a big year for our funny little digital friends. Not only are Facebook rolling out their Emotions function but in the last week the ‘tears of joy face’ emoji has been named as the Oxford Dictionary’s word of the year. They have been described as a more concise and more personable way of conveying a message or emotion. We are inclined to agree – but are they suitable for email marketing? Experian’s research showed that over 50% experienced an increase in open rates with the use of emojis in their subject lines which sounded very promising. We put it to the test by sending out three e-shots with and without emojis in the subject line.
How we set it up the e-shot test
We can choose a custom list of contacts to which we would send out these tests. Our database is relatively small so we decided to use everyone on our contact list to see if we could get an overall view of the reaction to emojis. We were then able to choose the amount of people we would send the A/B test to and the length of time the test would run until the the type with the highest open rate would then be sent out to the rest of our contacts. We sided with 50% of our contacts at random for the A/B test which equated to 1500 contacts per email type. A further 3000 would be sent out to the other 50% using the most successful subject line.
Caveat: this was by no means a perfect test. In an ideal world we would send out these tests to a number of demographics to further segment the data from the e-shot and remove as many variables as possible. Regardless, it showed some interesting results.
Spoiler alert: if you don’t want to read the statistics here is a little summary. Overall the emoji subject did not fare well. Not badly, but in terms of open rate success, it lagged over 2% on average less than the subject line without emojis. For a debrief on why this might have happened scroll down past the test stats!
The stats
Test 1
Sent on a Wednesday afternoon at 14:35 we had one subject line with no emojis and one with two emojis (at the beginning and at the end). Within 15 minutes the subject line without emojis had gained a 2.5% lead in open rates, stretching this out to over 4% in the subsequent 2 hours and 10 minutes. For all of the below graphs the blue line represents the emoji subject line and the red line represents the text based subject line.
Test 2The second test was somewhat hampered by a technical issues which meant that over 65% of the emails were not delivered. With our email campaign schedule dictated by our own content creation strategy we therefore had use this incomplete send as test 2 and the remainder as test 3. For test 2 we decided to use the same format with the emoji subject line with one emoji at the beginning and one at the end. We also decided to send it mid morning in order to see if our dreary eyed recipients would have their days lightened by an amusingly emojified subject line! Different emojis were used this time for the two emojis and initially this seemed to have a more positive impact with the emoji subject line holding a 1-1.5% increase in open rate for the first 40 minutes or so. The end result though was a close run contest that resulted in a draw between the two subject lines after a longer test period of over 4 hours.Test 3Using the contacts that had failed to receive the emails from the second test we switched up the format of the emoji subject line only using one emoji at the beginning. Overall the email with the emoji subject line lagged behind by around 2% open rate for the majority of the 2 hour and 20 minute test, only closing the gap to within 0.5% in the final 15 minutes.