Split testing email, part 1

Yesterday I introduced you to Charles Farnes-Barnes, a novice internet entrepreneur who's just begun tentatively exploring the possibilities of split testing. Charles used a simple split test to establish that a landing page improved the performance of his newsletter, and he was so taken with the results that he's now thinking about other ways to employ the technique.

He thinks about his newsletter. At the beginning of every one he puts a featured item about the new lines in stock, under the same headline, "This week's fashion selection". Perhaps that wording is not very compelling? What about "Fresh in this week: hot new fashions!" or "Stand out from the crowd in the latest gear!"? He decides to try out these alternatives using a split test.

You'll remember that when Charles did the landing page test for his discount offer he used discount codes so he could measure which of the two variants was more successful. This time however he can't use the same technique because the headline relates to items that anyone can buy in his shop. How will he be able to tell which customers came from the newsletter, and also which version of the newsletter they received? He's stumped, so he goes to talk to his web designer, Hank Ubernerd.

Hank is a bright chap so he comes up with an answer right away: multiple landing pages. He'll create one landing page for each headline that Charles wants to test. All the landing pages will look exactly the same, except they'll each have a different URL (web address). He'll be able to tell Charles at the end of the week exactly how many customers came to each one, and hence which headline was the most effective.

How will Hank track the number of visitors? There's a number of different ways he can do this but since the hosting service for Charles's web site lets him download his site's access log, he chooses to use that in conjunction with a log analyzer program to identify how many different visitors each landing page will get. There are a lot of different log analyzers on the market but Hank likes Mach5 Fast Stats Analyzer for its flexibility and number of reports. A great alternative for visitor tracking if you can't get hold of the access log is to use Google Analytics, which is an entirely free service and easy to set up (though not guaranteed to be as accurate as a log file analyzer). I'll cover using Google Analytics for tracking purposes in more detail in a later post.

Now all Charles has to do is create his three different newsletters: one with the old headline and the others with the two variant headlines he's going to test against it. Each one is linked to its respective landing page. He takes his email address list and ... uhhhh ... he realises he needs to randomize it for the test. Fortunately Hank is able to help again this time by creating a simple Excel spreadsheet that shuffles the entries. (I'm aware there are better ways to solve this problem but Charles and Hank don't - yet).

Now at last Charles is all set. He sends his newsletters out to his three lists and waits for the results...

What happened next? Wait and see!