Measuring Success of Website Testing

I recently wrote a post about some of the common challenges I’ve encountered running testing programs for my clients. I’d like to expand on one of them – selecting KPIs. I was reading a post the other day about the three Cardinal Sins of KPIs and Performance Metrics[i] and realized that many eCommerce managers make the mistake of defaulting to their primary business KPIs as the measure of success for every test. While the goal of testing is ultimately to impact these business KPIs, it is important to recognize how each step in the process impacts the whole.

When selecting KPIs for your business, it makes sense to focus on things that impact the bottom line – what’s my end goal conversion rate and AOV, how many users are finding products and adding them to the cart, how many users are signing up for email or registering, etc. These metrics can tell you important information about the health of your business, and by measuring them regularly over time you can get an early read when things are changing, or when your own changes significantly impact your business.

For testing, selecting KPIs is very different. Just like your business has a main goal and specific metrics that can tell you how you’re performing against your goal, each test you run should have a clear goal and associated metrics. Sometimes, the main goal of your test may have nothing to do with your primary website metrics. For example, one of my clients recently ran a test optimizing their careers page because they needed to get more workers in the field – could you imagine judging that test on it’s impact to service leads instead of applications submitted?

While the above example should be fairly obvious to most, other situations are more nuanced. Take homepage tests, for example. The goal of your homepage should be to get users to another page on your site as fast as possible. The likelihood of changes to your homepage having a significant impact on your bottom line is small, but if you measure the impact to either a category or product page, you are likely to make a noticeable impact. The problem with going from the homepage straight to the final conversion metric is that there are too many decision points in the middle. If your test reduces bounce rate from the homepage and gets users looking at more products, that’s a win even if it doesn’t result in purchases. Your takeaway from this test shouldn’t be that you didn’t have an impact on orders, it should be to find where you lost the users you’ve pushed further down the funnel and optimize the new drop-off point.

To ensure that you are not selecting KPIs that are too far from your test, I like to break the larger conversion funnel into several smaller funnels. At a minimum, every eCommerce site has at least two flows – landing page to cart, and cart to confirmed order. For checkout, I actually prefer to separate the cart as its own single goal page – get users into the checkout flow – then look at the next page through order complete as a separate flow. The important thing here is that you choose success metrics for your tests that can be directly impacted by your changes, then optimize each successive step in the flow as you drive more users to it. This is known as the waterfall approach to testing. Ultimately, the flows you look at will depend upon your unique business.

[i] https://www.linkedin.com/pulse/3-cardinal-sins-kpis-performance-metrics-bernard-marr