Internal crawl errors and unnecessary redirects affect user experience and crawlability – thus your rankings. Checking for these issues is an important part of reviewing past SEO efforts to ensure they are up-to-par with modern search engine optimization techniques.

Crawl Errors: How to Assess

There are a few ways to check your site for internal crawl errors. The most thorough approach is to use multiple resources and tools, such as the three below.

3 Ways to Check Your Site’s Crawl Errors

1. Google Search Console (formerly known as Google Webmaster Tools): The crawl errors report in GSC has multiple tabs, but the one to focus on here is the Not Found tab (see screenshot below). This report contains URLs that Googlebot has determined as non-existent (404 error code).

Graph with site URL errors

You can get more information by clicking an individual URL. In doing so, you will see a popup window with information, such as first detected and last crawled (Error details) tab. If the URL is found in a sitemap, you’ll see the “In sitemaps” tab and a link to the sitemap containing the bad URL. The “Linked from” tab will show you pages on the site that were found to be linking to the 404 page.

error detail

Action Items

1) Ensure your sitemaps do not contain URLs returning a 404 error
2) Remove all internal links to 404 pages
3) In many cases, URLs returning a 404 should be 301-redirected to the most related/similar page on the site. If there is no similar or related page, it is OK to let it return a 404 error. Executing No. 1 and No. 2 above should ensure the URL does not get crawled in the future. The exception is if there are quality external links pointing to the page returning a 404. If this is the case, the URL should be redirected to retain link equity.
4) After completing steps 1-3, use the “Mark as Fixed” feature in the Crawl Errors report. Then, check back periodically to see if the issues have truly been fixed or if new crawl errors are found.
5) To force Google to “see” that redirects have been put in place, use the Fetch as Google First enter the URL, click “Fetch” then click “Submit to Index.” This will force Google to recrawl the URL and it will see the 301 directive. You can submit up to 500 individual URLs per month with this feature.

 

2. Bing Webmaster Tools: Similar to GSC, Bing Webmaster Tools has a Crawl Information report that helps you identify pages that have returned various status codes.The 404 errors show up in the “tab” labeled 400-499 (Request errors).

Site crawl information

Action Items

1) Ensure your sitemaps do not contain URLs returning a 404 error
2) Remove all internal links to 404 pages
3) In many cases, URLs returning a 404 should be 301-redirected to the most related / similar page on the site. If there is no similar or related page, it is OK to let it return a 404 error. Executing No. 1 and No. 2 above should ensure the URL does not get crawled in the future. The exception is if there are quality external links pointing to the page returning a 404. If this is the case, the URL should be redirected to retain link equity.

 

3. Screaming Frog SEO Spider: Screaming Frog can be used for free for up to 500 URLs. If you have more URLs, you’ll have to purchase the tool. The above two methods are sufficient to identify, monitor and solve the majority of, if not all, issues with crawl errors. If you would like another method, check out this post on the Screaming Frog site, How to Find Broken Links Using Screaming Frog.

Redirect Chains: How to Assess

Over the lifetime of a website, multiple migrations, architecture changes, and changes in products and services may occur. Many reasons that pages will have been redirected. In some cases, sites end up with redirect chains — when a URL redirects to another URL that is redirecting to a third URL. In an ideal world, URLs being redirected should only redirect once. As a marketer, it is important to understand this and know how to identify whether redirect chains occur.

Thankfully, Screaming Frog makes this easy. If you are going through a site migration, you can follow the process here, How to Audit Redirects Using Screaming Frog.

If you are not going through a migration, there still may be redirects in place on the site. An easy way to look for redirect chains is by looking in the .htaccess file and using the method in the above article (upload URLs in list mode) to verify the redirects are pointing to the right place. If you are on a Windows server, there’s a good chance you are using the URL Rewrite Module. You should be able to find a list of all URLs being redirected in your web.config file. Then again, use the same method above to look for redirect chains.

These fixes to broken URLs and redirects can have a huge impact of your bottom line. If you’re ready to dig even deeper into your eCommerce site’s SEO health you can access our free workbook for a thorough assessment.

New Call-to-action