Category Archive a b testing


Speed Is King

In today’s hyper-wired world of satellites, and internet: speed is king. I once heard it said that 1 year on the internet is equal to 7 years offline. Nowadays, I think it’s more like 10 years or more.

And the harsh reality is that, as the online space evolves with increasing rapidity, what works well on your website today may not be very effective in a few months’ time.

That’s why we’ve developed a set of “Testing Results Protocols” in our businesses – to help speed up our split-tests and make improvements more quickly.

First of all, I couldn’t stand waiting for “statistical validity”. (In a minute, I’ll explain why I think that concept just slows down the growth of your business to a crawl. I’ll also give you quite a few advanced shortcuts you can use to help speed up your testing results too).

Plus, I’ll show you some new upgrades in our split-testing software that makes setting up these tests faster and quicker too.

But first let’s look at…

The 3 Levels of Testing Results

1.    Donuts (zeros)
2.    Incremental Improvements (also known as “whispers”)
3.    Breakthroughs (also known as “screams”)


Rule #1: You Can’t Multiply Donuts


Let’s do some math together:

0 X 10 = zero
0 X 100 = zero
0 X 10,000 = zero
0 X a million = ZERO

In other words, you can’t multiply donuts (zeros)!

Here’s a rule of thumb: If you set up a sales letter test and 300 people have seen your letter and no one has bought – STOP THE TEST IMMEDIATELY.

Basically, it means that your sales letter is not working, and it’s time to move on. (Personally, I would even stop the test after 150 people saw the letter because I want to find a winner as quickly as possible.)

This is not the time to get despondent either. It’s actually great news. Because you want to find out what’s NOT working as quickly as possible. That way you can move on to the next test and find the next big breakthrough.

NOTE: There are exceptions to this rule… For example, when you’re selling expensive equipment, courses or packages. If you’re selling a $10,000 piece of construction equipment – then maybe a 0.3% conversion rate is acceptable.

However, if you’re selling anything below $100, as a general rule, don’t accept conversion rates below 0.3%.


Rule #2: Don’t Wait Too Long For Incremental Improvements


Rule #2 is similar to rule #1. The bottom line is: you need to gather information and results quickly.

Why? Because you’re wasting time if you’re not aggressively looking for the ‘big breakthroughs’ in your business.

Let’s take some advice from the world’s greatest investor, Warren Buffet: Mr. Buffet makes investments based on ‘opportunity cost’. This means he doesn’t look at an opportunity and think “I can make 20% return if I invest 1 million dollars in that company.”

Instead, he looks at all of the other opportunities he’s “missing out on” by NOT investing the million dollars somewhere else.

And here’s how this applies to your split-testing strategy:

Scenario #1: The “normal” way

Let’s imagine you set up a split test. And the 2 factors are initially producing very similar results. They are “nose-to-nose”. But after 60 days you finally identify a clear winner that gives you a 10% improvement in your conversion rate. That’s not bad.

However: what did you LOSE by waiting so long for statistical validity to kick in?

Scenario #2: The “faster” way

You set up the same test as in ‘scenario 1’. However, you abandon the test after 1 week because it wasn’t a ‘breakthrough’ result – in other words, the results were too close and the test failed to produce a clear winner.

Next, you set up a new test, and you get a 25% improvement in your conversion rate after only 2 weeks. And then you set up another test and you get another 15% improvement after 2 weeks.

So with this strategy, you achieved a 40%+ improvement in 5 weeks, instead of the 10% in 8 weeks.

The point to get across here is this: by abandoning tests that are “nose-to-nose” quickly, you’ll find the ‘big breakthroughs’ faster.


Rule #3: Become A Relentless Hunter Of “Big Breakthroughs”


This is a concept I stumbled upon by being impatient. Basically you want to look for big results fast.

The Big Breakthrough Formula:

You have a ‘Big Breakthrough’ when…

The Winner – Square root of the winner > the losing element

The big breakthrough formula is simple… You subtract the “square root of the winner” from “the winner”, and the number you’re left with should be BIGGER than “the loser”.

This will become a lot more clear in the example below. It sounds more complicated than it is…

Example of the Big Breakthrough Formula in action:

Let’s say you’re doing A-B split testing, and element “A” has 20 sales and “B” has 12 sales.

First, calculate the square root of “20” – which is 4.47. Now, simply plug-in the rest of the numbers:

20 (winner) – 4.47 (square root of 20) = 15.53

The next question is: Is 15.53 a bigger number than 12 (the number of sales of the “losing” element)?

In this case, the answer is ‘yes’, so we have a ‘big breakthrough’ result, and can move on to testing something else against this winning element.


An Answer to Hardcore Believers of Statistical Validity


Frankly, is this method perfect? Of course not. I would even say, by picking winners quickly – you’ll be wrong 10% of the time or so.

In the grand scheme of things, is this important though?

I’ll take SPEED over accuracy any day of the week. Even if I am wrong 20% of the time, the increase in my conversion rates will quickly make up for it.

The reality is, the results will self-correct over time as you test more variables. For example, let’s say I end a split test early and I was wrong about the winner… I’m going to set up a NEW test – and hopefully I’ll crush the “wrong winner”. And I’ll set up another test and beat the champ again… and again… and again…

To recap: focus on achieving BIG breakthroughs with your testing. Don’t be afraid to end a test early. You do NOT need 200 results to have a valid test. I think that’s just insane (especially on a sales letter).