TL;DR

Crawling and indexing issues can prevent important pages from appearing in search results and limit your website’s visibility. MMC Digital’s Crawling and Indexing Analysis service is an investigation into the root causes of discoverability problems through technical audits, log file analysis, and site structure reviews. 

The end result for clients is an actionable roadmap, a strategy that – when implemented – will help improve crawling, indexing, and overall search performance.

Pain points

As a website grows, one of the problems that can occur is that crawls are poorly directed, often resulting in a limited understanding of the information on the site and its importance. This can, in turn, lead to issues with having important pages on your site indexed. 

While issues with crawl budgets are often overstated (it should really only be of concern to extremely large ecommerce sites, news sites and the like), a site which has grown organically over time (and with minimal oversight) can end up sending mixed messages that result in crawling and indexing issues.

So, if you’re struggling with any of the following, get in touch:

• Trouble ranking or indexing new pages.
• A large number of ‘discovered – not currently indexed’ in Google Search Console.
• A large number of ‘crawled – not currently indexed’ pages showing on Google Search Console.

What we mean by crawling and indexing analysis

This is one of the most fundamental aspects of any SEO offering – but, honestly, by the time it becomes an issue, it can be so knotty that it requires a lot of untangling. For most of the brands we work with, we can take a quick look at GSC, robots.txt and sitemap.xml files and give a reasonably quick thumbs up, but if there are large portions of the site with discoverability or indexing issues, that’s where there can be a lot of analysis required.

Again, the length of time spent on this can differ – sometimes there are some simple robots.txt directives that are holding things up, or the sitemap is badly organised or populating incorrectly – these are things we’ll check out and can fix relatively quickly, and do so regularly during standard auditing, but when it goes beyond these things, we’re going to have to start pulling stuff apart and looking at the bits.

MMC Digital’s crawling and indexing analysis service

On the rare occasions that our clients are struggling with crawl and indexing issues that can’t be resolved quickly, we’ll look to analyse your log files (the raw data generally held by your hosting provider) and carry out some deep dives on your site structure, information hierarchy, content relevance, authority and look to establish a cause (or at least a range of potential causes) that can be combatted to improve your performance.

This work will result in a strategy document that you can either tackle yourself, that you can give to a third party for confirmation and execution, or that you can have us tackle ourselves. The choice is yours – when you brief us with this kind of work, our aim is just to make sure you have the information you need to overcome this problem. 

We’re not looking to sell you additional services or to claim we’re the only one that can solve your problem – we love what we do, and the problem solving aspect of this kind of analysis is hugely enjoyable. If you choose to trust us with implementation, that’s just a bonus!

What can our crawling and indexing analysis service do for your business?

The easiest way to summarise this is that it’s a detective service. The end result of this analysis project is not resolution, it’s a thorough understanding of your existing problems. What we try to provide is a roadmap to recovery that is implementable by whichever partner you choose to carry out the work, when a client comes to us for this specific service, we pop on our private detective fedoras and begin Philip Marlowe-ing our way through your online presence until we find the culprit.