Labrador - FAQ
Why are you visiting my site?
We occasionally run test crawls, over restricted domains (eg .ac.uk, or .uk). These
domains may includes your website. We try very hard to ensure that all protocols
are observed, including the robots.txt
guidelines. Additionally, we throttle our crawlers on a per-host basis to ensure that no
one server is being overloaded. If you wish to be removed from any future crawls, you can
contact the author, or update your robots.txt file accordingly.
What are the CPAN (Perl) requirements of running Labrador?
Labrador was built as a crawler to supplement Terrier.
A Labrador is a type of retriever canine - which we thought was appopriate, and in theme with terrier.