Split testing email, part 2: visits and hits

In my previous post you'll recall we left Charles Farnes-Barnes, our novice internet marketer, anxiously waiting for the results of his email test. He'd created a split test to compare three different variants of a headline in his email newsletter. For the sake of simplicity, we'll refer to them in this article as headlines A, B and C.

The method he'd chosen to evaluate the success of the headlines was to count visits to three different landing pages set up specifically for the test. Something Charles didn't know, but which his canny web expert Hank explained to him, was the difference between visits and hits.

Hits are what a lot of simple counters give you: the number of times a page has been fetched. It's not a very meaningful figure because if you set up a page which your devoted mother opens forty times every day because she has it as her home page, then you'll get forty hits a day even if nobody else ever looks at it. This is one reason amateur webmasters like hit counters: they tend to exaggerate the popularity of a site.

A visit on the other hand is a more scientific measure, in that the analytics software reporting it tries its best to filter out repeated hits from the same source. If it's doing its job, it'll just count your mother once - not very filial perhaps, but fair.

Fortunately the analytic software that Hank uses does count visits accurately, so now let's move on to look at the result he got in this case. The three lists were each taken from Charles's master list of 2,000 email addresses. Each list contained 666 or 667 entries taken at random.

The three results were: for list A, 10 visits; for list B, 80 visits; and for list C; 90 visits. Now, what can one conclude from these figures? Clearly headline A is a poor performer compared to B and C, but what about B compared to C? Can Charles conclude that headline C is better than B, or is the difference simply due to random factors?

In other words, is the difference between B and C significant? That's what we're going to look at in the next episode of this exciting saga, but beware, there's maths ahead!