The Epiphany Business Featured Image
I’ve heard over the years people say that having 50% of tests win is about average (that is, reach 95% statistical confidence). That means that half of all testing programs will lose more than half the time. I am not refuting this, as I can see how it happens.
I have also heard CROs I know and trust saying that +90% percent of their tests win.
It begs the question, “What is the difference between organizations that win most of their tests, and ones that lose most of their tests?” It’s a big question, especially considering the cost of testing.

The Cost of Testing

Before you do A/B testing, you should figure out the cost per test. Assume you will run 5 tests per month. It might look something like this:
  • Test Tool Cost (Monthly Cost of Tool/ 5 Tests) i.e. $800
  • Cost to Create Test Requirements (In a document or email)
  • Cost to develop and QA Test – i.e. 2.5 hours
  • Cost to admin and launch test – Estimated time 20 mins.
  • Cost to observe and report on test – Estimated 30 mins.
  • Cost to communicate results – Estimated 15 mins, with 5mins for each reader.
In my experience, a typical cost for running a test considering all the above would be about $1,300 for a typical eCommerce Business.
Keep in mind, the above does not include any costs around gaining the insight that lead everyone to think that the test was a good idea and would result in more sales.

Opportunity Costs

Just as important as the true cost, is the opportunity cost of running a test while holding off on another effort that could have yielded more sales. If your development, web, marketing, and product teams are spending time to test one thing, that’s less time that could have been spent on other work, including testing that could have produced better results.

Win More Often

Considering the costs, the only way to pay for a testing program is to win tests. The more tests win, the better you do as a company, since test winners are capturing sales that were out there to capture all along.

Be Compelled

One rule to any testing program is to test when you feel compelled, rather than testing for the sake of testing. Yes, you want good “velocity” in your testing program, but if you’re testing for the sake of testing, your test costs will exceed the benefits of making the few improvements you “guessed right”

Insights are Compelling

Nothing is more compelling than evidence. Gaining real insight into why your prospective customers are not buying can throw you and pretty much anyone who cares into high gear.
With insights, the test treatments are typically obvious to anyone with some eCommerce experience. Not only is the solution easier (when you know what your are trying to fix), but the chances of the test winning go up dramatically.

Insights take Time

There is a reason everyone does not test based on insights. It’s hard.
Gaining insights takes time, and thought and often preparation as well. It also becomes easier the more experience you get working with tools you use to gain insights.

How to Get Insights?

In order to gain insights, you have to get closer to the customer. So the question becomes, how to do that quickly and efficiently. When considering how to go about gaining insights, know that you’re looking for insights that are:
  1. High Quality
  2. Actionable
  3. Quick to Obtain

Qualitative leading Quantitative

Generally, you will have more success with getting good actionable insight from qualitative methods such as user testing, polling, and customer service interviews. There are of course exceptions, especially for the master analyst, but pretty much any master analyst will tell you that the majority of insights they garner from analytics can be done in less than an hour once they gain their bearings.
Here are some ways to gain insights that are pretty much industry standards. I am ordering by the level of success I’ve had with them over the past 15 years.

Qualitative Approaches

  • Support/Sales Interviews & Focus groups
  • User Testing (Recruited from site or segmented panel)
  • A/B Testing post-test analysis
  • User focus groups
  • Heuristics
  • Site feedback forms
  • Polling (tied to analytics)

Quantitative Approaches

  • On-Page Analytics (If the audience is segmented for better context)
  • Customer Journey Analysis (Segmented on New & Returning Users)
  • Audience Segmentation Analysis (requires advanced configuration)
  • Cart Abandonment Analysis
  • Post-Purchase Surveys

Epiphany Business - Ways to Gain Insights

Record your Patterns

There’s a lot of insight to be gained by benchmarking a lot of your user behavior metrics. This is base level information that will guide you along your insights gathering, allowing you to be more informed and put your discoveries into greater context. Below are things I always do/update or review before going on a mission for insights.
  • What is the device mix?
  • What % of users are returning?
  • Main User Paths (for Purchase, Non-Purchase, New and Returning visitor segments)
  • How long it takes for 95% of people to purchase after their 1st visit.
  • What percentage of visits are at each phase of the buying cycle?
  • How people scan key pages according to mouse move heatmaps.

Isolate Behaviour

If you look at the above list, you likely recognize the need to isolate behavior. The good news is that through understanding the business and the site, you can define some behavior as being common for a particular persona. Once you have a few of these, you can create a custom segment and use it as a reasonable proxy for that persona.
For example, if you’re looking for women who are in the consideration phase of shopping for dresses than you likely only need to look in analytics for people who have visited more than 2 dress category pages, stayed on the site for more than 90 seconds, OR visited the site more than once. This segment will be “qualified” and can be a proxy for analyzing that segment.
The key to isolating behaviour is to cut through the noise and see what people are doing, not what the data is doing.
Once you have these segments, you can easily apply them to test data and see if that “winner” was also a winner for women who show intention to buy dresses.


Want to make more money? Then improve the customer’s experience. Want to know more about the customer experience? Get insights. Want Insights? Do the work (and I hope this article helped make that a bit easier).