Google’s Spiders – The Basic SEO Guide

If you run a website then you’ll understand the importance of your business appearing high in the Google search results for your chosen keywords. Achieving that depends on having the right search engine optimisation (SEO) strategy. We recently came across an infographic from Quicksprout explaining how Google crawls and indexes webpages. We’ve used it as the basis for our basic SEO guide, explaining how to make sure your site features in Google search results.
 

What is Google’s index?

Google’s index is the list of all the pages on the internet that Google knows about and that appear in its search results. In order for pages on your website to appear in the results they need to be part of the index. Only around 10% of all the pages on the internet actually appear in the index. The rest are either blocked by webmasters or don’t appear at all. This is usually due to technical error or a misunderstanding on the part of the person running the website. If your website doesn’t make the index then you’re missing out on a huge amount of potential traffic on the internet.
 

How do sites get added to the index?

Google uses computer programmes known as spiders to crawl the internet, looking for new pages and updating its records on the pages that are already there. Crawls can take place several times a day or as little as every six months. How often a webpage is crawled depends in part on its PageRank and also on the number of other websites that link to your page. Remember that simply being crawled regularly won’t get your site higher up the Google search results so hundreds of links pointing to your website from all over the internet won’t help boost your ranking.
 

Why can’t Google find my website?

If your website still isn’t appearing in Google’s search results after several months and you’ve tried submitting your website to Google manually that suggests that there is a problem with your website’s configuration. The most common errors are covered in the infographic.  An incorrectly configured Robots.txt file is the equivalent to putting up a big closed sign outside your shop. If Google’s spiders think you don’t want them to index your website then they won’t index it.
If your host is unreliable then Google is less likely to feature your page. This happens both because the spiders are likely to encounter problems trying to access the website and because Google places a penalty on these websites.
If you’ve previously used Blackhat SEO techniques and attempted to artificially boost your site’s place in the Google ranking then Google may de-index your website. If you’re using Google Analytics then you will have received a message about this in Google Webmaster tools. Fixing this problem will involve unpicking a lot of previous bad practice but it’s essential if you want your website to appear in the results again.
 
If you’d like more information on defining a good search engine optimisation strategy then visit our SEO page. You can use active social media profiles, blogs and other strategies to boost your website’s freshness and place in the index. We’d be delighted to help you achieve that.