Voiced by Amazon Polly

We’re often asked by clients to help them understand the impact of a site change or redesign. More often than not, they want to measure impact using conversion rate. This is understandable, but the way most people look at conversion rate leads to incorrect conclusions.

The main issue is this: Conversion rate changes over time are far more influenced by traffic channel mix than by improvements to the site. This means you could redesign (or as we’d suggest, run a continuous conversion optimization program on) your site, only to have it look like the conversion rate dropped — when in reality, you have improved the conversion rate of your website. To avoid this error you need to measure conversion rate by channel, seasonality, device, and more. 

This scenario is separate from the fact that many redesigns (especially those done for aesthetic purposes only) end up making the website worse. In this article, I’ll be discussing changes that make a site better but which are hidden by traffic channel mix changes.

If you’d like our eCommerce conversion rate team to help evaluate your conversion rate by channel, seasonality, and device, or evaluate the impact of a site change on conversion rate, you can learn more or reach out here: Inflow’s Conversion Rate Optimization Service.

Case Study 1: Site Updates That Appear to Have Lowered Conversion

In this example, we did a conversion optimization project for a small startup. After our recommended changes had been implemented, the following chart makes it look like the conversion rate was better before the changes, or at least not affected by them:

Users and eCommerce Conversion Rate

There are a few issues here that make this chart inaccurate in terms of judging the efficacy of the site updates:

  • There’s no seasonality data. Since this is a startup, we have no accurate history to compare against to get year-over-year data. This is particularly problematic because the startup operates in a highly seasonal vertical.
  • This top-level conversion rate chart doesn’t show traffic volume, and changes in that metric can dramatically affect the conversion rate.
  • This chart also doesn’t show the traffic channel mix, which (as already mentioned) is by far the most important factor for time based conversion rate analysis.

Seasonality

Seasonality is key. Most businesses have some degree of seasonality, with some that only operate at certain times of year. For any business, however, conversion rate over time needs to compare seasonally as well as sequentially. 

In this case, without previous year data, there’s no way to tell if the August conversion rate data was good or bad. However, since the product is related to a summertime sport, we can assume there is going to be a fall off as we head toward winter. Since many purchases for summertime activities happen in the spring, it’s reasonable to assume conversions and sales should start declining in the fall — meaning that conversion rate is strong for August.

Traffic Volume

This is related to traffic channel mix but is its own topic as well. If a business increases its traffic dramatically (whether via paid, organic, referral, direct, or other), it always experiences a decrease in conversion rate along with the increase in traffic

Why is this? The main theory is that such dramatic increases are results of increased exposure, but the increased exposure is not as targeted.

Think of it as being ranked for a bunch of new terms in SEO, such as being ranked for “toys” where before you were only ranked for your specific product, i.e. “Slinky.” 

I subscribe to this theory. There may be other theories about why this happens, but regardless of the reason, I can confirm this rule holds for every client we’ve ever had.

Traffic Channel Mix

This is the big one and the focus of this article. In our example, you can see the following shifts over time:

Traffic channel mix: Direct, Instagram, Google, FB

Instagram traffic had been declining prior to the updates as a percentage of total traffic, and it continued to decline after the updates. Instagram had been the best converting channel, and its decline dragged the overall conversion rate down. The other major dynamic is the rise in social CPC, which at its launch converted at only 0.25% (see chart below). This also dragged down the overall conversion rate.

Conversion rate by channel:

Conversion rate by channel: Instagram, direct, Google, FB

Here you can see that Instagram traffic converted at about the same level (if you add “I.instagram” and “instagram,” which should be done via a filter in Google Analytics) before and after the updates — but the mix dynamic discussed above means that its traffic decline would pull down the overall conversion rate. 

The launch of social CPC traffic (fb_ig) was an initial drag on the overall conversion rate, since it converted initially at only ~0.25%. As it ramped up in volume, the conversion rate increased. (An exception to the “as traffic increases, conversion rate decreases” theory is the launch of a new paid program or investments in SEO — where SEO had been neglected previously). 

This pulled the overall conversion toward itself — in this case, in August, it was pulling the overall conversion rate closer to 1%, which is below where it was before the updates.

So what about the site updates — were they good or not? 

To answer that, I’d look for the traffic sources that were stable in terms of volume. In the chart above, you see that those are “direct” and “google.” Direct is a black box — so it’s not the best candidate. However, Google is one of the harder acquisition channels to budge (with the exceptions noted above). 

In this case, Google organic traffic volume stayed about the same throughout the updates. Looking at its conversion rate, we see it go up in the 2 months following the site updates. Therefore, we can say the site improvements worked for Google and probably also worked for the other channels (though that affect is hidden by the changes in that traffic).

Was that a satisfying answer to upper management? Nope, but it is an accurate reflection of how the website is actually doing in terms of converting users. It also shows why not evaluating conversion rate by channel is a serious mistake that can hide valuable information. 

Case Study 2: Seasonality & Traffic Volume Issues

Another one of our clients revamped their mobile smartphone experience prior to engaging with us. They asked for our analysis of the change, since their executives were not impressed with the before-and-after results.

Here is the top-line average conversion rate over time by device — with mobile not showing much improvement after February/March 2017, when the changes were made. One thing to note is that their peak season was usually in November/December, which is most visible on the desktop line:

Top-line average conversion rate over time by device: Desktop, mobile, tablet

For the same time period, we see a huge shift toward mobile, partly due to their now-SEO-friendly site, and partly due to the global shift to mobile devices. (Prior to that, they had an older demographic and were behind the curve in terms of transition to mobile.)

Top-line average conversion rate over time by device: Desktop, mobile, tablet

Since we had strong seasonality, the key analysis needed to be around year-over-year changes. Here you can see year-over-year eCommerce conversion rate by device. 

Note that of the top 6 months’ strongest year-over-year performance for this time period, 4 of them occurred after or during the redesign. This is enough to show that the redesign was a success — but the client was still not satisfied with this explanation.

eCommerce Conversion % Difference from Previous Year (All)

So we dug into conversion rate by channel. Without burdening you with all the different charts we looked at, we found that the key dynamic for mobile was organic. That large increase in mobile traffic volume above was driven by organic. 

That in itself is fantastic — organic traffic increased 2.5x in the period after the redesign. But, as stated before, this followed the rule that as traffic increases dramatically, the conversion rate decreases. In this case, organic conversion rate declined significantly year-over-year. 

eCommerce Conversion % Difference from Previous Year (Organic)

So the redesign wasn’t successful after all, right? 

Not exactly. Since you can’t pay for a trip to Hawaii with conversion rate, we look to revenue, since it includes both volume and conversion. Mobile year-over-year revenue was the best it’s ever been due to the dramatic increase in organic traffic volume, even though the conversion rate went down dramatically:

Revenue Difference Year Over Year

Parting Example: Interference Caused by Display Ad Traffic

Do those examples seem overly complicated? Consider the poster child for conversion rate destruction: Display. 

Here’s the scenario: 

You launch Google’s pay-per-conversion Display traffic program, in which you only pay if Google’s algorithms get you a conversion.

Google sends 100,000 new visitors to your site, and one of them converted.

Your conversion rate tanks!

Are you better or worse off? I’d say you’re better off by one conversion and by the revenue associated with it (unless your bonus depends on conversion rate). This underscores the fact that, while you should track conversion rate, it’s merely an indicator, not a measure of success.

If you’d like our eCommerce conversion rate team to help evaluate your conversion rate by channel, seasonality, and device, or evaluate the impact of a site change on conversion rate, you can learn more or reach out here: Inflow’s Conversion Rate Optimization Service.