Website Spiderability Test

What does it do?

Test whether or not a website can be spidered, i.e. explored automatically by search engines and other automated programs. This is one of the most important tests of all.

Why is it important?

Spidering is the process by which a computer program (such as WebScan, Google or Yahoo) can discover the pages in a website – it reads one webpage, looks for pages that page links to, before reading those pages, and so on. If for any reason a website cannot be spidered, it will suffer enormously:

  • It can’t be explored by search engines, and will usually not appear in them (or appear lower than they should).
  • It will usually be totally inaccessible, as most software to help the visually impaired or disabled is unable to penetrate the website. In many countries, this could make the website illegal.

Sites that are non-spiderable should fix this as their first (if only) SEO priority.

How is it measured?

WebScan has a more advanced spider than most, with the unusual capability of exploring many traditionally 'non-spiderable' sites. In addition, WebScan knows about any manual pages which you have instructed it to explore manually.

This means that (paradoxically) WebScan can usually identify non-spiderable content via use of it's own spider. Whenever a page is accessed via a means that a traditional spider would not manage, that route is marked as non-spiderable. Pages that can only be accessed via a non-spiderable route are used to calculate the volume of the site which is non-spiderable.

The score is calculated based on the proportion of spiderable pages, with a perfect 10 being awarded to sites where every page is spiderable.

Potential problems

My site doesn't need to be spiderable. If you're testing a site that is intentionally non-spiderable, e.g. behind a login area, this test is redundant. Exclude the test manually to circumvent this issue.

How to improve this score

Ensure the website can be spidered. Depending on the technology you use this could be a trivial problem, or an extremely complex and involved one.

How to use this test effectively

This test should return a perfect score, or very close to it. If this score is low, it should be immediately addressed as your first priority.

Impact on Summary Scores

Summary Score Percentage Penalty
Accessibility Score 9.4% Penalty if below 7.8
Marketing Score 3.9% Penalty if below 7.8
Overall Score 3.0% Penalty if below 2.2
Technology Score 6.1% Penalty if below 7.8