Timing your tests is just one of the critical factors involved in getting accurate results. I created a checklist that covers the timing and participant factors and much more.
Get the full check list right here!
Let’s look at two very important and closely related items on the checklist and break down the answers:
• Are results being looked at within full-week cycles?
• Was the test given enough time for participants to purchase?
It’s all about time: run a test long enough
Whatever the reason, not running a test long enough will rob you of the truth, so you might as well have not run the test in the first place.
A test should be run for at least a full seven-day cycle PLUS the length of the buying cycle. This is due to purchase behavior being different during different days of the week. While daily variations are different for each site, there is normally a weekly pattern.
For instance: Let’s say your site sells children’s bicycles, and your analytics tell you that 14 percent of purchases occur between four and seven days after the user’s first visit to the site. From analysis, it’s discovered that a significant amount of Monday purchases are from weekend research, and Tuesday and Wednesday purchases have a high degree of purchases from same-day visitors (who are trying to get the product by the weekend). Therefore, any test that starts on Monday and ends on Thursday will be biased toward the test variation that is preferred by the spontaneous visitor (i.e. homepage highlight of bikes less than $100). However, this may not really be a winner if the rest of the weekday’s results were included (where purchases are more likely looking for quality and may be turned off by a “discount site”).
It is advised that you let your test run for at least two weeks.
Run a test with enough participants and goal completions
A test needs people, and the more people the better. A good way to judge whether or not the test has enough participants is by looking at the number of goals achieved (i.e. purchases). You should never ask, “How many people are needed in the test?” but rather, “How many conversions does the test need?”
Naturally, the more variations a test has, the more participants it will need. Consider 100 conversions per variation to be a MINIMUM, and only after the test has run long enough (as mentioned above) and other criteria have been met (see below).
Use 7-day cycles
When testing, you most likely have to test against a full week cycle. This is because people often behave differently during different parts of the week.
For instance: If your site sells toys for small children, your site’s reality might be that a lot of research traffic occurs on the weekend when the children are available for questioning (i.e. “Hey Ty, what’s the coolest toy in the world these days?”). Another reality for a toy site might be that often the “Add to Cart” button does not need to get hit until Tuesday evening, given a lot of toys are not needed until the weekend when birthday parties are typically held. Consider this and now ask yourself if a test run from Wednesday through Sunday (five full days with lots of data) is enough?
The reality is, almost every eCommerce site (from the more than 100 eCommerce analytics I’ve done test analysis on) has a seven-day cycle. You may have to figure out which days to start and stop, but it’s there.
Therefore, if you don’t use a seven-day cycle in your testing, your results are going to be weighted higher for one part of the week than another. Using the toy store example above, ask yourself how valid would a test that excluded Tuesdays be? What if the test was started on a Wednesday and runs 12 days to include two weekends and only one “best sales day” Tuesday?
These two items are just scratching the surface when it comes to making sure you’re doing everything you need when running a test. I highly recommend the full checklist to cover all aspects of your testing.
Don’t just stop at the checklist! To really understand the full context of the checklist and how it can best help you run accurate tests, download the free eBook on how to get the most accurate tested results.