As you know, I think split tests rock and you should definitely do them. However, over at TechCrunch Robert J. Moore brings up a great point about A/B testing:
…What if, like most start-ups, you only have a few thousand visitors per month? In these cases, testing small changes can invoke a form of analysis paralysis that prevents you from acting quickly.
Consider a site that has 10,000 visitors per month and has a 5 percent conversion rate. The table below shows how long it will take to run a “conclusive” test (95 percent confidence) based on how much the change impacts conversion rate.
If you’re a startup with low traffic, is his point, you don’t have as much opportunity to cycle through tests as might a site with more visitor flow, so you want to make sure the tests you do run will have a big impact. Change only something small about the home page, and you may find yourself needing to let the test run for weeks before you reach significance. Some implications of this:
- Take traffic into account when designing a test plan. If you’re doing a banner ad campaign with several different segments, it probably only makes sense to run tests in the segments big enough to get reasonable traffic. If you’ll only get a few conversions from a segment, you likely won’t have enough volume to generate significant results. In the principle of minimizing the amount of resources expended on projects, only spend time preparing and tracking tests if you are likely to see results.
- Start big, go small. If you’re in early days of testing start at the concept level– different tones, different layouts, different messages. Test things that are likely to have a bigger impact, and refine from there. That being said…
- Small changes can have big impacts. Once I tested search ads that were completely identical except that one had a period after the text and one didn’t. The period generated a surprisingly big lift. I just heard from a friend that changing one word in the call to action on their home page lifted response by 20%. On the other hand I’ve run tests where two ads were completely different and didn’t really get a significantly different result.
If you’ve got tiny traffic you will have fewer test cycles available to you. You don’t always know ahead of time what’s going to move the needle, so check your results regularly and end tests that don’t seem to be going anywhere.