inflow-logo

Inflow: eCommerce Marketing Agency

  • Services
    • SEO
    • PPC
    • CRO
    • Paid Social
  • Clients
    • Case Studies
  • About Us
    • Contact Us
  • Insights Blog
  • Request Proposal
  • Services
    • SEO
    • PPC
    • CRO
    • Paid Social
  • Clients
    • Case Studies
  • About Us
    • Contact Us
  • Insights Blog
  • Request Proposal

Home > eCommerce Digital Marketing Blog > Conversion Rate Optimization > A/B Testing > The “Test Window” Methodology for eCommerce Conversion Optimization Testing

The “Test Window” Methodology for eCommerce Conversion Optimization Testing

Posted By Keith Hagen on September 8, 2015

  • 4shares

When I work on our conversion optimization campaigns and I describe it in our latest eBook, Stop Wasting Your Time When Testing eCommerce Sites: How to Get the Most Accurate Tested Results, I use the methodology I call the “Test Window.”

This is how you will not only find the TRUE outcome of your testing but also gain insights to further drive optimizations and profit for weeks, months and quarters ahead.

The Test Window is a simple approach to reporting out on tests so that you can be more certain about which variation won and what degree of impact you are really looking at.

I do have to warn you that the Test Window methodology will be hard work, but it will be worth it as it yields more accurate and insightful conclusions. It will also make you question your test tool’s reports, which you should.

All About The “Test Window” Methodology

Based on the 3 Laws of Testing, which are outlined in the eBook, it may be obvious what the “Test Window” is and how to run your tests properly.

It’s easiest to think of the Test Window in terms of steps.The Test Window

Step 1: Only let new visitors into the test. This way returning visitors later in their purchase cycle will not skew results and potentially set the test off to a false start. Get as many people into the test as possible.

Step 2: Don’t look at the test for a full seven days. If you don’t have a statistical winner at this point (most test tools will tell you the test has reached 95 percent confidence), let the test run for another seven-day cycle and don’t peek.

Step 3: Turn off the test to new visitors once you have a statistical winner (at seven-day intervals). Turning off the test to new visitors will allow the participants already in the test to complete their buying cycle. Leave the test running for a full buying cycle.

Step 4: Report out on the test. To report out on the test’s overall results, you will simply look at your test tools test report. Now, because you used the Test Window, you will be able to believe the results because:

  1. Everyone in the test had a consistent site experience, spending it in the same test variation (no one seeing the control on a previous visit only to later experience a treatment).
  2. A full seven-day cycle was used so weekend days and weekdays were weighted realistically.
  3.  Every user (or 95 percent of them at least) was allowed to complete their buying cycle.

Test Window example: Below shows a test run using the Test Window. We only allowed first-time visitors into the test. You can see the first few days favored the control, which is common due to the control bias. (Even with limiting test participants to new visitors, there are still enough returning visitors from other devices, browsers, etc. to create a bias.) Once the test is turned off to new visitors and test participants are allowed to complete their buying cycle, the variation gains strength.

Test Window Results

Testing Properly eBook Call To Action
  • 4shares

0 Comments on "The “Test Window” Methodology for eCommerce Conversion Optimization Testing"

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Posts

  • Research and Analysis of the Top 50 eCommerce Brands’ Facebook Ads How do the top eCommerce companies use Facebook ads? We analyzed 50 top companies to find out what eCommerce Facebook ads they were running. Here are the trends we found.
  • How to Build An eCommerce Dashboard for KPI Reporting Using Google Sheets & Supermetrics While Dashboards in Google Analytics are nice features, they really lack some significant functionality, such as displaying the percent change between two numbers (say traffic last month […]
  • Google’s “First Link Counts” Rule: Why Your eCommerce Category Pages Aren’t Linking to Products Optimally Every once in awhile an age-old technical SEO issue comes back around to remind me that this type of optimization is as important as ever. It's been a long time since I've heard […]
Keith Hagen

Keith Hagen

Keith Hagen comes from the web optimization world, and has been gaining insights into what is and isn’t working with websites since 2002.

View Author’s Profile

Related Categories

  • A/B Testing (11)
  • eCommerce Page CRO (10)
  • Mobile Conversion Optimization (5)
  • Tools and Plugins (14)
  • Usability (5)
  • Most Popular Posts:

    eCommerce Marketing Automations Systems Compared
    Technical Mobile Best Practices for SEO and Usability
    Expanding the Horizons of eCommerce Content Strategy
    Thin & Duplicate Content: eCommerce SEO
    5 Ways eCommerce Content Audits Can Increase Revenue
    Want to get content like this straight to your inbox? Subscribe to our weekly content alerts and monthly Inflow Insights newsletter now.

    Categories

    • SEO
      • Content Marketing
      • Link Building
      • On-Page SEO
      • SEO Strategy
      • Technical SEO
    • Paid Advertising
      • Goal Metrics and Analytics
      • Paid Search
      • Paid Search Shopping
      • Paid Social
    • Conversion Rate Optimization
      • A/B Testing
      • eCommerce Page CRO
      • Mobile Conversion Optimization
      • Tools and Plugins
      • Usability
    • Case Studies
    • eCommerce Strategy
      • KPIs and Reporting
    • Digital Marketing Trends in eCommerce
    • Inflow News

    Request a Proposal

    We'll build a custom proposal to meet your goals. Get the process started now.

    Google Premier PartnerInflow is a facebook-certified-creative-strategy-professional Moz Recommended Company Inc 5000 Inflow Clutch Profile
    • Services
      • SEO
      • PPC
      • Conversion
    • Case Studies
      • SEO
      • PPC
      • Conversion
    • Insights Blog
    • Resources
    • More
      • Contact
      • Careers
      • Press Info
      • Privacy Policy
    REQUEST A PROPOSAL
     
    CALL US AT 303-905-1504
    Monday - Friday, 8 a.m. - 6 p.m. (MST)
     
    facebook twitter linked-in linked-in rss

    Send this to a friend