TL;DR

Back in the long ago, log file analysis was one of the only methods to track interactions with your site – it’s a record of every interaction and response made, and it allows you to see a lot more than a quick glance at one of the files would appear. Although it’s not used as often these days – partly because, until Screaming Frog built some incredible tools, it was a pain in the backside, but with every visit to your site, your server records a timestamp, the requesting server’s IP, the URL requested, the status code returned and so much more, that ignoring can mean ignoring a treasure trove of data. 

Log file analysis tells us the relationship between search engine bot frequency and page performance, the existence of status errors, overlooked or uncrawled content, site performance and much more – that’s why we still do it.

Pain points

If you’re having issues with any of the following, our audits may be what you need:

• Problems that don’t seem to have been solved by best practice
• Unexplainable drops in performance
• Visibility issues on specific device types or browsers
• Trouble ranking new pages

What we mean by log file analysis

One of the exceptionally few benefits of getting old is that we’ve seen most of the issues that can occur in a site’s log file, and while we always try to standardise our approach with documentation, checklists and the like, we still pore over every log file like it’s our first. This is a hugely rich source of data and we don’t like to miss a thing. That’s what analysis means to us – bringing our experience and existing knowledge, but still examining everything as though for the first time, making sure we investigate every nook and cranny for potential issues that might be costing you money.

MMC Digital’s log file analysis service

Clients choosing MMC Digital for log file analysis, you can expect us to look at the following:

• Crawled URLs: checking exactly which URLs search bots are able to access (and, importantly, which ones they ignore) to help us understand how often key pages are crawled and how bot behaviour maps to your site structure.
• Crawl frequency: looking for which bots are active on your site, how frequently they visit, and how crawl patterns evolve over time, which helps to see where search engines are focusing their attention.
• Broken links and errors: to discover inappropriate response codes, crawl errors and broken paths encountered by crawlers, so you can resolve technical issues before they impact visibility or UX.
• Redirects: to identify temporary and permanent redirects as they’re encountered by search bots (including possible discrepancies between real bot behaviour and what traditional browser-based tools might report).
• Crawl budget: not often an issue outside of extremely large sites, but in those cases we can analyse which areas of your site receive the most (and least) crawl activity to reduce wasted resources, improve efficiency, and help search engines prioritise the pages that matter commercially.
• Large and slow pages: to review download size and response times, spot performance bottlenecks, heavy templates, or oversized assets that could slow down crawling and impact indexing.
• Uncrawled and orphan pages: to reveal orphaned pages (inaccessible without external link), overlooked content, or important URLs that search engines haven’t discovered yet.

What can log file analysis do for your business?

Log file analysis offers a wide scope for research and analysis, and can contribute to improving the visibility and usefulness of a website. In addition, as it’s no longer a universally practiced technique, the chances are that a business regularly analysing their log files is going to catch issues their competitors don’t, making it more likely that you’ll be better prepared for success in search.