Code Documentation

Labrador - FAQ

Why are you visiting my site?
We occasionally run test crawls, over restricted domains (eg .ac.uk, or .uk). These domains may includes your website. We try very hard to ensure that all protocols are observed, including the robots.txt guidelines. Additionally, we throttle our crawlers on a per-host basis to ensure that no one server is being overloaded. If you wish to be removed from any future crawls, you can contact the author, or update your robots.txt file accordingly.

What are the CPAN (Perl) requirements of running Labrador?

  • LWP::Bundle
  • HTML::Parser
  • HTML::LinkExtor
  • HTML::LinkExtractor
  • DelayLine

Why Labrador?
Labrador was built as a crawler to supplement Terrier. A Labrador is a type of retriever canine - which we thought was appopriate, and in theme with terrier.