inflow-logo

Inflow: eCommerce Marketing Agency

  • Services
    • SEO
    • PPC
    • CRO
    • Paid Social
  • Clients
    • Case Studies
  • About Us
    • Contact Us
  • Insights Blog
    • Resources
  • Request Proposal
  • Services
    • SEO
    • PPC
    • CRO
    • Paid Social
  • Clients
    • Case Studies
  • About Us
    • Contact Us
  • Insights Blog
    • Resources
  • Request Proposal

Home > eCommerce Digital Marketing Blog > SEO > Log File Analysis for SEO – Primer & How-To

Log File Analysis for SEO – Primer & How-To

Posted By Glen McMurry (Alumni) on October 25, 2016

  • 11shares

What is Log File Analysis for SEO?

Log file analysis has many applications outside of SEO, such as site security. In terms of search engine optimization, the process usually involves downloading the file from your server and importing it into a log file analysis tool, where all the information about every “hit” on the site (whether bot or human) can be analyzed to inform SEO decisions and learn about previously unknown issues.

Log file analysis is an arduous process that frequently results in the discovery of critical technical SEO problems that could be found no other way. Log files contain incredibly accurate data that allow a brand to better understand how search engines are crawling their site and the kind of information they are finding.

Log file data includes a record of the URL/resource that was requested, action taken, time and date, IP of the machine it originated from, user agent/browser type, and other pieces of information.

 

log file example

A new server log entry like the one above will be created each time a resource is requested from your website.

 

One benefit of log file analysis for SEO is to audit where a site’s crawl budget is spent. The higher the authority of the domain, the higher the crawl budget that is allocated by the search engines. While log file analysis can’t impact the crawl budget your site receives from Google, it can help to optimize how that budget is used. Some of the ways this is done include:

  • Identifying which URLs are crawled the most frequently and optimizing toward those
  • Identifying client and server errors and implementing fixes
  • Identifying large/slow URLs and making efforts to increase speed
  • Identifying orphaned pages that would not show up in a site crawl

 

Log File Analysis Steps

Because of its large size, log file analysis has always been difficult. There are products out there to make it easier, such as Screaming Frog’s new log file analysis tool, Logz.io and Google’s BigQuery solution, but it is still a long project.

The general process is below, with steps 3 and 4 being the most time intensive:

  1. Begin analysis of data
  2. Filter by user-agent to identify crawl bots
  3. If interested in larger trends, pull multiple days – this will almost certainly mean that a third-party tool is necessary, such as Splunk or Logstash.
    • If the desire is to work with Excel, convert the file to a .csv, take a random sample of ~100,000 rows of data, use text-to-columns to format data appropriately, and label rows for ease of use
  4. Pull a day’s worth of data if only interested in a snapshot in time
  5. Export log entries in common log file format from the server
    • Filter by request type to find most/least crawled URLs
    • Filter by HTTP response codes to find status errors
    • Evaluate crawl waste using a combination of all columns
    • Etc.
  6. Optional: Expand database
    • Import data from Analytics, site crawls, Google Search Console and potentially the CMS
  7. Determine issues
    • Look for errors to fix, redirects chains to shorten, orphaned pages that should be linked to or deleted, rogue bots not obeying the robots.txt file

 

Do You Need to Analyze Your Log Files?

Analyze files

The importance of a log file analysis depends on how mature a website is in its SEO efforts. As previously stated, the level of effort involved in a log file analysis is very high. The effort expended from the person or team doing the analysis is similar to a Content Audit. There are often easier, more pressing issues to address than those that require a log file analysis.

If, however, most of the basic SEO best practices are in place and you’re concerned about site speed, crawl budget and the depth or frequency of various search engine crawlers, log file analysis will be an illuminating project worth performing.

  • 11shares

0 Comments on "Log File Analysis for SEO – Primer & How-To"

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Posts

  • Using an Innovation Litmus Test to Find a Great eCommerce Digital Marketing Agency Finding a good eCommerce agency isn't easy. Learn how to use our Innovation Litmus Test to separate the good agencies from the bad.
  • How to Hire an eCommerce Director (Plus 3 Questions to Ask During the Interview) Trying to figure out how to hire an eCommerce director for your business? Here's my perspective as the CEO of an eCommerce-focused digital marketing agency on what to look for.
  • 5 Content Strategies to Improve Your eCommerce Business Understanding the need for creating content on your website, and driving traffic, is one thing. Having that content meet your bottom-line business goals, however, is a bigger challenge […]
Glen McMurry

Glen McMurry (Alumni)

With more than six years of experience in Inbound Marketing, McMurry has a great track record in delivering well-rounded inbound strategies that impact all aspects of the marketing mix. Particularly specializing in international SEO, he has worked on a wide range of clients in finance, travel, telecoms, FMCGs and more.

View Author’s Profile

Related Categories

  • Content Marketing (13)
  • Link Building (16)
  • On-Page SEO (14)
  • SEO Strategy (11)
  • Technical SEO (35)
  • Most Popular Posts:

    eCommerce Marketing Automations Systems Compared
    Technical Mobile Best Practices for SEO and Usability
    Expanding the Horizons of eCommerce Content Strategy
    Thin & Duplicate Content: eCommerce SEO
    5 Ways eCommerce Content Audits Can Increase Revenue
    Want to get content like this straight to your inbox? Subscribe to our weekly content alerts and monthly Inflow Insights newsletter now.

    Categories

    • SEO
      • Content Marketing
      • Link Building
      • On-Page SEO
      • SEO Strategy
      • Technical SEO
    • Paid Advertising
      • Goal Metrics and Analytics
      • Paid Search
      • Paid Search Shopping
      • Paid Social
    • Conversion Rate Optimization
      • A/B Testing
      • eCommerce Page CRO
      • Mobile Conversion Optimization
      • Tools and Plugins
      • Usability
    • Case Studies
    • eCommerce Strategy
      • KPIs and Reporting
    • Digital Marketing Trends in eCommerce
    • Inflow News

    Request a Proposal

    We'll build a custom proposal to meet your goals. Get the process started now.

    Google Premier PartnerInflow is a facebook-certified-creative-strategy-professional Moz Recommended Company Inc 5000 Inflow Clutch Profile
    • Services
      • SEO
      • PPC
      • Conversion
    • Insights
      • Insights Blog
      • Case Studies
      • Resources
    • More
      • Contact
      • Careers
    •  
      • Press Info
      • Privacy Policy
    REQUEST A PROPOSAL
     
    CALL US AT 303-905-1504
    Monday - Friday, 8 a.m. - 6 p.m. (MST)
     
    facebook twitter linked-in linked-in rss

    Send this to a friend