When I work on our conversion optimization campaigns and I describe it in our latest eBook, Stop Wasting Your Time When Testing eCommerce Sites: How to Get the Most Accurate Tested Results, I use the methodology I call the “Test Window.”
This is how you will not only find the TRUE outcome of your testing but also gain insights to further drive optimizations and profit for weeks, months and quarters ahead.
The Test Window is a simple approach to reporting out on tests so that you can be more certain about which variation won and what degree of impact you are really looking at.
I do have to warn you that the Test Window methodology will be hard work, but it will be worth it as it yields more accurate and insightful conclusions. It will also make you question your test tool’s reports, which you should.
All About The “Test Window” Methodology
Based on the 3 Laws of Testing, which are outlined in the eBook, it may be obvious what the “Test Window” is and how to run your tests properly.
It’s easiest to think of the Test Window in terms of steps.
Step 1: Only let new visitors into the test. This way returning visitors later in their purchase cycle will not skew results and potentially set the test off to a false start. Get as many people into the test as possible.
Step 2: Don’t look at the test for a full seven days. If you don’t have a statistical winner at this point (most test tools will tell you the test has reached 95 percent confidence), let the test run for another seven-day cycle and don’t peek.
Step 3: Turn off the test to new visitors once you have a statistical winner (at seven-day intervals). Turning off the test to new visitors will allow the participants already in the test to complete their buying cycle. Leave the test running for a full buying cycle.
Step 4: Report out on the test. To report out on the test’s overall results, you will simply look at your test tools test report. Now, because you used the Test Window, you will be able to believe the results because:
- Everyone in the test had a consistent site experience, spending it in the same test variation (no one seeing the control on a previous visit only to later experience a treatment).
- A full seven-day cycle was used so weekend days and weekdays were weighted realistically.
- Every user (or 95 percent of them at least) was allowed to complete their buying cycle.
Test Window example: Below shows a test run using the Test Window. We only allowed first-time visitors into the test. You can see the first few days favored the control, which is common due to the control bias. (Even with limiting test participants to new visitors, there are still enough returning visitors from other devices, browsers, etc. to create a bias.) Once the test is turned off to new visitors and test participants are allowed to complete their buying cycle, the variation gains strength.
