How Much Traffic Do I Need To Start Doing A/B Testing?

I’d like to start measuring improvements quantitatively, but I don’t think I have enough traffic. What’s the point when I can start doing real A/B Testing?

Improving a site’s effectiveness is a guessing game, that is, until you have enough traffic to demonstrate a difference between two versions. Version A, version B, that’s where A/B Testing (or split-testing), comes from.

When do you know you have enough traffic to make that measurement count?

You need a lot of traffic, but if you think you can make a big difference, you don't need as much.

If Version B Makes a Bigger Difference, You Need Fewer Visits

Optimizely, a popular A/B Testing tool, comes up with these numbers in their online sample size calculator. (Note: Optimizely bases their numbers on data from the A/B tests they run for everyone, and so other online sample size calculators will give you different numbers.)

You’ll need 26,000 visits, or 13,000 for each version, if you’re currently converting 3% of your visitors and you think you can increase that by 20%. See the calculation on Optimizely with a 20% increase.

But you’ll only need 10,600 visits, or 5,300 for each version, if you’re currently converting 3% of your visitors and you think you can increase that by 30% instead. See the calculation on Optimizely with a 30% increase.

Screenshot of the Optimizely Sample Size Calculator

That difference comes from the fact that a bigger difference between two versions will take a smaller sample size to measure.

You might ask:

How can I reach a 30% conversion rate (or more) without trying all the nasty tricks that people pull on visitors for them to convert.

Well, it requires that you think deeply about helping the visitor make some progress.

Be on the lookout in the next few weeks for articles on how to do that, and a breakdown of how to to calculate whether a Version B will be better.

Stay Sharp!


Get articles like this one, delivered on Friday.

To learn to sharpen your own stuff.

Coming in the next few weeks: we’ll be talking about sharpening your freelancing business.

And here's a list of the past articles to get a sense of what you'll get.

Plus, receive a link to a video of a presentation I gave explaining the Jobs-To-Be-Done theory

Another option: on Twitter (@pascallaliberte), you'll get notified of new articles just the same, just a few days later.
With the email list, you get it first.