A/B Cross Device Testing Flaws for CRO


2 Comments
  1. This is terrific. Do you have any recommendations on how I can implement geotargeted testing using Google Optimize? Or another testing platform that you would recommend to do this? Thanks.

    • Thanks for reading PA. This kind of test is a bit tricky to set up, which is why it is still very uncommon. Most test tools do not facilitate this out of the box – they are designed to randomize groups, not push people into pre-assigned groups based on your criteria, geo location included.

      We are able to do this using custom JS code that we build for every test we run and our own Geo Location script we keep on our server. If you are not working with a testing company or are in a less sophisticated organization that doesn’t have resources to do this, you likely will have to hack together your results as outlined below.

      Most AB testing tools do allow geo targeting for a split test (ie: only allow people from these locations into the test), they just don’t allow you to define the audiences for each individual version based on region. If you are using one of the standard tools like VWO or Optimizely you likely will have to set up 2 separate “tests” and set up your two groups as the geo targets for the 2 tests. You will then “test” only 1 version, meaning you’re basically using the tool to do a content serve to 2 different regions. With Google Optimize now doing some personalization as well, you might also be able to leverage that to serve the two different versions rather than setting up 2 tests.

      In the end, regardless of whether you set up multiple “tests” or use personalization targeting, you’ll have to manually do a lift and confidence calculation to see how different the results are for your two regions instead of leveraging the built-in confidence calculations from your tool. There are plenty of these calculators out there for free, just do a quick search for “AB test confidence calculator”.

      Hope this helps, and happy testing!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About The Author

A photograph of Keith Hagen next to a bike on a road next to a river.

Keith Hagen (Alumni)

Keith Hagen comes from the web optimization world, and has been gaining insights into what is and isn’t working with websites since 2002.

View Author’s Profile

Request a Proposal

Send this to a friend