net.yacy.crawler

Interface Balancer

All Known Implementing Classes:

HostBalancer, HostQueue, LegacyBalancer

public interface Balancer

Method Summary

Methods

Modifier and Type	Method and Description
void	clear() delete all urls from the stack
void	close() close the balancer object
Request	get(byte[] urlhash) get one url from the crawl stack
java.util.Map<java.lang.String,java.lang.Integer[]>	getDomainStackHosts(RobotsTxt robots) get a list of domains that are currently maintained as domain stacks
java.util.List<Request>	getDomainStackReferences(java.lang.String host, int maxcount, long maxtime) get lists of crawl request entries for a specific host
boolean	getExceed134217727()
int	getOnDemandLimit()
boolean	has(byte[] urlhashb) check if given url hash is contained in the balancer stack
boolean	isEmpty() check if stack is empty
java.util.Iterator<Request>	iterator() iterate through all requests in the queue
Request	pop(boolean delay, CrawlSwitchboard cs, RobotsTxt robots) get the next entry in this crawl queue in such a way that the domain access time delta is maximized and always above the given minimum delay time.
java.lang.String	push(Request entry, CrawlProfile profile, RobotsTxt robots) push a crawl request on the balancer stack
int	remove(HandleSet urlHashes)
int	removeAllByHostHashes(java.util.Set<java.lang.String> hosthashes) delete all urls which are stored for given host hashes
int	removeAllByProfileHandle(java.lang.String profileHandle, long timeout) delete all urls from the stack by given profile handle
int	size() get the size of the stack

Method Detail

close

void close()

close the balancer object

clear

void clear()

delete all urls from the stack

get

Request get(byte[] urlhash)
            throws java.io.IOException

get one url from the crawl stack

Parameters:

urlhash -

Returns:

the request for an url by given url hash

Throws:

java.io.IOException

removeAllByProfileHandle

int removeAllByProfileHandle(java.lang.String profileHandle,
                           long timeout)
                             throws java.io.IOException,
                                    SpaceExceededException

delete all urls from the stack by given profile handle

Parameters:

profileHandle -

timeout -

Returns:

the number of removed urls

Throws:

java.io.IOException

SpaceExceededException

removeAllByHostHashes

int removeAllByHostHashes(java.util.Set<java.lang.String> hosthashes)

delete all urls which are stored for given host hashes

Parameters:

hosthashes -

Returns:

number of deleted urls

remove

int remove(HandleSet urlHashes)
           throws java.io.IOException

Parameters:

urlHashes, - a list of hashes that shall be removed

Returns:

number of entries that had been removed

Throws:

java.io.IOException

has

boolean has(byte[] urlhashb)

check if given url hash is contained in the balancer stack

Parameters:

urlhashb -

Returns:

true if the url is queued here, false otherwise

size

int size()

get the size of the stack

Returns:

the number of urls waiting to be loaded

getOnDemandLimit

int getOnDemandLimit()

getExceed134217727

boolean getExceed134217727()

isEmpty

boolean isEmpty()

check if stack is empty

Returns:

true iff size() == 0

push

java.lang.String push(Request entry,
                    CrawlProfile profile,
                    RobotsTxt robots)
                      throws java.io.IOException,
                             SpaceExceededException

push a crawl request on the balancer stack

Parameters:

entry -

Returns:

null if this was successful or a String explaining what went wrong in case of an error

Throws:

java.io.IOException

SpaceExceededException

getDomainStackHosts

java.util.Map<java.lang.String,java.lang.Integer[]> getDomainStackHosts(RobotsTxt robots)

get a list of domains that are currently maintained as domain stacks

Returns:

a map of clear text strings of host names to an integer array: {the size of the domain stack, guessed delta waiting time}

getDomainStackReferences

java.util.List<Request> getDomainStackReferences(java.lang.String host,
                                               int maxcount,
                                               long maxtime)

get lists of crawl request entries for a specific host

Parameters:

host -

maxcount -

maxtime -

Returns:

a list of crawl loader requests

pop

Request pop(boolean delay,
          CrawlSwitchboard cs,
          RobotsTxt robots)
            throws java.io.IOException

get the next entry in this crawl queue in such a way that the domain access time delta is maximized and always above the given minimum delay time. An additional delay time is computed using the robots.txt crawl-delay time which is always respected. In case the minimum time cannot ensured, this method pauses the necessary time until the url is released and returned as CrawlEntry object. In case that a profile for the computed Entry does not exist, null is returned

Parameters:

delay - true if the requester demands forced delays using explicit thread sleep

profile -

Returns:

a url in a CrawlEntry object

Throws:

java.io.IOException

SpaceExceededException

iterator

java.util.Iterator<Request> iterator()
                                     throws java.io.IOException

iterate through all requests in the queue

Returns:

Throws:

java.io.IOException