The Startup Gardener Rotating Header Image

A/B, It’s Easy As 123

If you are anything like me, when you are purchasing a product you want some evidence that the product is the best for your budget. You’ll consult at reviews, your magic 8-ball, customer feedback, forums etc. Yet when it comes to making a change to something that has the potential to have a big impact on customer experience (thus the bottom line) we often make choices without really knowing if our choice is the best option out there. Doesn’t sound like the best way to do things does it.

Testing, Testing, One, Two, Three…

So what do you do when it comes time to evaluate the comparative effectiveness of subtleties of your marketing1? Well, one of the simplest techniques is A/B Testing (also known as bucket or split testing).

Dora Gen Y

Will the new Dora appeal to the key Gen Y market? A/B Testing do your thang! (source)

Wikipedia tells us that A/B Testing “is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples“. Put simply, with A/B testing you are comparing variations of something to see which is the most effective. For example you could do A/B Testing on your advertising by creating a number of different advertisements, each slightly different. Then you gauge the conversion rate of the advertisements. You can then keep the best then rinse and repeat if necessary. Another example where A/B testing would be useful would be when making changes to your customer conversion page, should the buy now button go on the left, right, top or bottom of the page? Should it be a red button or blue? Should you charge $29.99 or $52.50?2 All of these things can be put to the test.

Lies, Damn Lies and Statistics…

Now how do you decide which option is the winner? Obviously he one with the biggest result, right? Well, not quite. We also need to know if the results have some significance are just the result of chance. To do it properly lets turn to our friend the science of statistics for the answer.

Dilbert. Uncomfortably close to real life.

Dilbert. Uncomfortably close to real life.

What we are looking for is statistical significance. How do you go about this? Well it involves a simple bit of maths that is explained nicely in post by Jason Cohen entitled “Easy statistics for AdWords A/B testing, and hamsters”. So check it out and then you and me both can avoid the trap of letting our gut feel lead us astray when really it’s just the bad shellfish you had last night.

The Measure of Success

How does one implement a test like this? The first thing to decide is what is the definition of a successful transaction. If might be clicking a certain link, maybe it is money changing hands, maybe it is subscribing to a newsletter. Often the signal for these events occurring in a web based business is when a certain page is loaded. Anyway, once you know what success is you can work out ways to count it. If your testing your online advertising conversion rates most advert serving services provide tools for tracking conversion rates. For google adwords (and most others) it “involves inserting a small snippet of code into specific pages on your website that indicate an important event has occurred, for example, the order confirmation page you show after a purchase“. If you’re trying to test the effectiveness of the position of your buy now button it is going to be a bit more involved. Needless to say (so why am I saying it?) you have the option to roll your own solution or use the services of one of the myriad of software providers who specialise in the business of a/b testing. What you should do really depends on your current businesses situation, the discussion of which I’ll have leave for another time as it is now time go, so…

Hasta Luego my lovelies,

-Jimmy McGrath
1. in this context marketing really includes anything that has an impact on your customers e.g. advertising language, product features, site layout, copy, colour scheme etc.
2. But tread with great care here as people don’t really like to find out you’re charging them more than somebody else…

Done any A/B testing recently? Let us know what you found out by leaving a comment on this post.

Don't be greedy, pass it on:
  • Print
  • Digg
  • Facebook
  • Google Bookmarks
  • HackerNews
  • MySpace
  • RSS
  • Slashdot
  • StumbleUpon
  • Twitter
  • Yahoo! Bookmarks


  1. Cromwell says:

    Jimmy, I haven’t done any A/B testing myself, but your post has opened my eyes to evidence of it on the web. For instance on‘s front page there is a big orange signup button, looking at the link behind the button you can see “ref=bigassorange”. It seems to me that they are testing how many signups they get with various types of button.

  2. Jimmy McGrath says:

    Good spot Cromwell! I wonder if you were to go to on a different day, (or from a different computer), would you see any changes to the button? Perhaps the the colour would change? Or maybe the size?