How Much Traffic Do I Need To Start Doing A/B Testing?
I’d like to start measuring improvements quantitatively, but I don’t think I have enough traffic. What’s the point when I can start doing real A/B Testing?
Improving a site’s effectiveness is a guessing game, that is, until you have enough traffic to demonstrate a difference between two versions. Version A, version B, that’s where A/B Testing (or split-testing), comes from.
When do you know you have enough traffic to make that measurement count?
You need a lot of traffic, but if you think you can make a big difference, you don't need as much.
If Version B Makes a Bigger Difference, You Need Fewer Visits
Optimizely, a popular A/B Testing tool, comes up with these numbers in their online sample size calculator. (Note: Optimizely bases their numbers on data from the A/B tests they run for everyone, and so other online sample size calculators will give you different numbers.)
You’ll need 26,000 visits, or 13,000 for each version, if you’re currently converting 3% of your visitors and you think you can increase that by 20%. See the calculation on Optimizely with a 20% increase.
But you’ll only need 10,600 visits, or 5,300 for each version, if you’re currently converting 3% of your visitors and you think you can increase that by 30% instead. See the calculation on Optimizely with a 30% increase.
That difference comes from the fact that a bigger difference between two versions will take a smaller sample size to measure.
You might ask:
How can I reach a 30% conversion rate (or more) without trying all the nasty tricks that people pull on visitors for them to convert.
Well, it requires that you think deeply about helping the visitor make some progress.
Be on the lookout in the next few weeks for articles on how to do that, and a breakdown of how to to calculate whether a Version B will be better.
Stay Sharp!
—
Pascal Laliberté
@pascallaliberte