Search engines send out what are called spiders, crawlers or robots to visit your site and gather web pages. These robots leave traces behind in your access logs, just as an ordinary person does. If you know what to look for, you can tell when a spider has come to call. That can save you worrying that you haven't been visited. You can tell exactly what a robot has recorded or failed to record. You can also spot robots that may be making a large number of requests, which can affect your page impression statistics or even burden your server.

Here you will find a list of spiders, so that you can easily identify them and also some more resources to help you spot these little blighters.

Route to Market