inflow-logo

Inflow: eCommerce Marketing Agency

  • Services
    • SEO
    • PPC
    • CRO
    • Paid Social
  • Clients
    • Case Studies
  • About Us
    • Contact Us
  • Insights Blog
    • Resources
  • Request Proposal
  • Services
    • SEO
    • PPC
    • CRO
    • Paid Social
  • Clients
    • Case Studies
  • About Us
    • Contact Us
  • Insights Blog
    • Resources
  • Request Proposal

Home > eCommerce Digital Marketing Blog > SEO > Technical SEO > The CRUFT Crusher (5 of 9 Winning Plays) – Identify & Remove Low-Quality URLs from Google’s Index

The CRUFT Crusher (5 of 9 Winning Plays) – Identify & Remove Low-Quality URLs from Google’s Index

Posted By The Inflow Team on July 11, 2017

  • 4shares

Inflow - Winning Plays - The Cruft Crusher Featured Image

Performing a content audit to determine which pages should be improved and which should be removed/pruned from Google’s index is an effective and scalable SEO tactic that Inflow has written extensively about in publications like Moz, the Oracle blog, and blogs by Kevy, Kapost, Shopify, Lemonstand, BigCommerce, and more. Part of our eCommerce Marketing Playbook, the Cruft Crusher play is an SEO tactic that uses the content audit process to remove cruft from Google’s index, thereby improving the overall quality of the domain, as well as internal linking factors like PageRank distribution.

About the Play

The word “cruft” is a term used by developers to describe superfluous code. In this case, cruft is applied to URLs indexed by search engines. It is quite common for us to see a very long tail of indexed URLs that provide little or no value. It is also likely that many of these could be doing more harm than good to your search engine rankings. The Cruft Crusher will help you do the analysis necessary to make informed decisions about which URLs to safely de-index.

Who’s It For?

SEO Professionals
Tech-Savvy Content Marketers

When to Use It:

This play is a safe, effective bet that nearly always works on large eCommerce websites. The results are easy to track, and it is easily rolled back in the unlikely event that you would want to do that. However, best results come from implementing the Cruft Crusher on domains will thousands or more pages indexed by Google.

How to Execute The Cruft Crusher

  1. Crawl all indexable URLs
    • This is your content inventory
    • We typically use Screaming Frog
  2. Gather additional metrics for each URL
    • URL Profiler is a good tool for this
    • SuperMetrics plugin for Excel or Google Sheets is another great option
  3. Put it all into a dashboard and begin analysis
    • This is your content audit
  4. Create a column in the spreadsheet that allows cell input only in these formats:
    • Leave As-Is
    • Improve
    • Remove
    Winning Plays - The CRUFT Crusher - Screenshot

    At this point, you should be viewing something similar to the spreadsheet below:

  5. Sort by landing page visits from organic search
    • Verify that your data is accurate by manually reviewing (spot checking) in your Analytics program (e.g., Google Analytics).
  6. Mark any URL with extremely few or no visits from organic search within the last year as a candidate for immediate removal from Google’s index
  7. Provide these instructions:
    • The Robots meta tag in the HTML header of these pages should read as follows:
      <META NAME=”ROBOTS” CONTENT=”NOINDEX,FOLLOW”>
  8. Check post implementation for the following common mistakes:
    • Use the browser’s search function in Source View or Chrome Inspect mode and type: <meta name=”robots”
      1. See if there is more than one result on the page for that search
      2. Technically speaking (though messy and not recommended), there can be multiple robots meta tags, but there should only be a single directive on whether to index or noindex, and a single directive on whether to follow or nofollow
    • It is unlikely, but you should also check the HTTP header response for an X-Robots-Tag that gives conflicting directives
  9. Once implemented, notate the Analytics timeline for easy validation later
  10. After about a month, you should be seeing data in Google Search Console indicating a reduction in pages indexed, while also seeing an improvement in other metrics.
Winning Plays - The CRUFT Crusher - Screenshot 2
This Inflow client saw a 31 percent increase in organic search traffic, and a 28 percent increase in revenue after pruning thousands of their lowest-quality product pages from Google’s index. Click the image to view the case study.

The Cruft Crusher Resources:

How to Do a Content Audit
Content Audit Toolkit Download
Cut the Cruft, MozTalk Denver
Most Effective Way to Improve Sitewide Rankings
Content Audit Strategies for Common Scenarios

  • 4shares

5 Comments on "The CRUFT Crusher (5 of 9 Winning Plays) – Identify & Remove Low-Quality URLs from Google’s Index"
  1. Ethan says:
    July 11, 2017 at

    Awesome post! The steps you’ve outlined make this process very actionable.

    What’s the best way to get a tag on certain product pages if I’m not able to add via the admin panel in BigCommerce?

    Ready to get the pruning underway. 🙂

    Reply
    • Mike Belasco says:
      July 11, 2017 at

      Hi Ethan,
      In a pinch you could try adding the tag via Google Tag Manager OR you can use a noindex directive in the robots.txt file (different from blocking the page). Here are links to articles about each option https://moz.com/blog/seo-changes-using-google-tag-manager https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/

      Cheers,
      Mike

      Reply
  2. Ethan says:
    July 12, 2017 at

    Ah-ha! These are very helpful. Thank you, Mike!

    Reply
  3. Emmerey Rose says:
    July 17, 2017 at

    Wow! This was pretty interesting Mike. Thanks for sharing! I was wondering, how is it different from disavowing links?

    Reply
    • Everett Sizemore says:
      July 27, 2017 at

      This is for internal content Emmerey, whereas disavowing links is done on external links.

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Posts

  • Case Study: Consistent Conversion Testing Leads to 90% Lift Over time, Google has increasingly shown preference for sites that provide users with a good experience, whether they're on desktop or mobile. Making sure your site is responsive across a […]
  • Why Content Quality Matters for Your eCommerce Website Do you want to keep the traffic from search engines coming to your eCommerce website? Competing search results along with Google’s periodic updates can cause organic traffic and […]
  • Facebook Ads for Furniture eCommerce Site Case Study: ROAS From 0 to 29.5 in 3 Months We’ll share the exact process we followed to increase a furniture eCommerce site’s ROAS from 0 to 29.5 in just 3 months. Roughly, for every dollar spent on their Facebook ads campaign, […]
Inflow Team Icon

The Inflow Team

This article was written by a member of the past or present Inflow team.

View Author’s Profile

Related Categories

  • Content Marketing (13)
  • Link Building (16)
  • On-Page SEO (14)
  • SEO Strategy (11)
  • Technical SEO (35)
  • Most Popular Posts:

    eCommerce Marketing Automations Systems Compared
    Technical Mobile Best Practices for SEO and Usability
    Expanding the Horizons of eCommerce Content Strategy
    Thin & Duplicate Content: eCommerce SEO
    5 Ways eCommerce Content Audits Can Increase Revenue
    Want to get content like this straight to your inbox? Subscribe to our weekly content alerts and monthly Inflow Insights newsletter now.

    Categories

    • SEO
      • Content Marketing
      • Link Building
      • On-Page SEO
      • SEO Strategy
      • Technical SEO
    • Paid Advertising
      • Goal Metrics and Analytics
      • Paid Search
      • Paid Search Shopping
      • Paid Social
    • Conversion Rate Optimization
      • A/B Testing
      • eCommerce Page CRO
      • Mobile Conversion Optimization
      • Tools and Plugins
      • Usability
    • Case Studies
    • eCommerce Strategy
      • KPIs and Reporting
    • Digital Marketing Trends in eCommerce
    • Inflow News

    Request a Proposal

    We'll build a custom proposal to meet your goals. Get the process started now.

    Google Premier PartnerInflow is a facebook-certified-creative-strategy-professional Moz Recommended Company Inc 5000 Inflow Clutch Profile
    • Services
      • SEO
      • PPC
      • Conversion
    • Insights
      • Insights Blog
      • Case Studies
      • Resources
    • More
      • Contact
      • Careers
    •  
      • Press Info
      • Privacy Policy
    REQUEST A PROPOSAL
     
    CALL US AT 303-905-1504
    Monday - Friday, 8 a.m. - 6 p.m. (MST)
     
    facebook twitter linked-in linked-in rss

    Send this to a friend