NAME

Labrador::Dispatcher::CrawlerAlloc::Partitioned

DESCRIPTION

A partitioning Crawler Allocator. URLs are allocated to crawlers based on the partition they fall in. This is done using Labrador::Common::Partition::* modules.

METHODS

init()
get_urls($self, $crawler_name, $count)

Returns an array of $count URLs that are in the partition named $crawler_name.

update_queues

Move URLs from URLAlloc to here.

register_crawler($crawler)

This crawler may now have partitions assigned to it.

unregister_crawler($crawler)

De-assign partitions for $crawler

queue_sizes

Returns the total size of the crawler queues

get_stats()

Returns some statistics about partitions.

REVISION

	$Revision: 1.5 $