net.yacy.crawler

Class RecrawlBusyThread

java.lang.Object

java.lang.Thread

net.yacy.kelondro.workflow.AbstractThread

net.yacy.kelondro.workflow.AbstractBusyThread

net.yacy.crawler.RecrawlBusyThread

All Implemented Interfaces:

java.lang.Runnable, BusyThread, WorkflowThread

public class RecrawlBusyThread
extends AbstractBusyThread

Selects documents by a query from the local index and feeds the found urls to the crawler to recrawl the documents. This is intended to keep the index up-to-date Currently the doucments are selected by expired fresh_date_dt field an added to the crawler in smaller chunks (see chunksize) as long as no other crawl is runnin.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

java.lang.Thread.State, java.lang.Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field and Description
private int	chunksize
private int	chunkstart
private java.lang.String	currentQuery
private boolean	includefailed
(package private) Switchboard	sb
private java.lang.String	solrSortBy
static java.lang.String	THREAD_NAME
long	urlsfound
private java.util.Set<DigestURL>	urlstack

Fields inherited from class net.yacy.kelondro.workflow.AbstractThread

busytime, memuse, running

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
RecrawlBusyThread(Switchboard xsb)

Method Summary

Methods

Modifier and Type	Method and Description
private boolean	feedToCrawler() feed urls to the local crawler (Switchboard.addToCrawler() is not used here, as there existing urls are always skiped)
void	freemem() is called when an outOfMemoryCycle is performed this method should try to free some memory, so that the job can be executed
boolean	getIncludeFailed()
int	getJobCount()
java.lang.String	getQuery()
boolean	job() Process query and hand over urls to the crawler
private boolean	processSingleQuery() Selects documents to recrawl the urls
void	setIncludeFailed(boolean includefailedurls) Flag to include failed urls (httpstatus_i <> 200) if true -> currentQuery is used as is, if false -> the term " AND (httpstatus_i:200)" is appended to currentQuery
void	setQuery(java.lang.String q, boolean includefailedurls) Set the query to select documents to recrawl and resets the counter to start a fresh query loop

Methods inherited from class net.yacy.kelondro.workflow.AbstractBusyThread

getBusyCycles, getBusySleep, getHighCPUCycles, getIdleCycles, getIdleSleep, getOutOfMemoryCycles, getSleepTime, intermission, notifyThread, run, setBusySleep, setIdleSleep, setLoadPreReqisite, setMemPreReqisite, setObeyIntermission, setStartupSleep

Methods inherited from class net.yacy.kelondro.workflow.AbstractThread

announceThreadBlockApply, announceThreadBlockRelease, close, getBlockTime, getExecTime, getLongDescription, getMemoryUse, getMonitorURL, getShortDescription, jobExceptionHandler, open, setDescription, shutdownInProgress, terminate

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface net.yacy.kelondro.workflow.WorkflowThread

close, getBlockTime, getExecTime, getLongDescription, getMemoryUse, getMonitorURL, getShortDescription, isAlive, jobExceptionHandler, open, setDescription, shutdownInProgress, start, terminate

Field Detail

THREAD_NAME

public static final java.lang.String THREAD_NAME

See Also:

Constant Field Values

currentQuery

private java.lang.String currentQuery

includefailed

private boolean includefailed

chunkstart

private int chunkstart

chunksize

private final int chunksize

sb

final Switchboard sb

urlstack

private final java.util.Set<DigestURL> urlstack

urlsfound

public long urlsfound

solrSortBy

private java.lang.String solrSortBy

Constructor Detail

RecrawlBusyThread

public RecrawlBusyThread(Switchboard xsb)

Method Detail

setQuery

public void setQuery(java.lang.String q,
            boolean includefailedurls)

Set the query to select documents to recrawl and resets the counter to start a fresh query loop

Parameters:

q - select query

includefailedurls - true=all http status docs are recrawled, false=httpstatus=200 docs are recrawled

getQuery

public java.lang.String getQuery()

setIncludeFailed

public void setIncludeFailed(boolean includefailedurls)

Flag to include failed urls (httpstatus_i <> 200) if true -> currentQuery is used as is, if false -> the term " AND (httpstatus_i:200)" is appended to currentQuery

Parameters:

includefailedurls -

getIncludeFailed

public boolean getIncludeFailed()

feedToCrawler

private boolean feedToCrawler()

feed urls to the local crawler (Switchboard.addToCrawler() is not used here, as there existing urls are always skiped)

Returns:

true if urls were added/accepted to the crawler

job

public boolean job()

Process query and hand over urls to the crawler

Returns:

true if something processed

processSingleQuery

private boolean processSingleQuery()

Selects documents to recrawl the urls

Returns:

true if query has more results

getJobCount

public int getJobCount()

freemem

public void freemem()

Description copied from interface: BusyThread

is called when an outOfMemoryCycle is performed this method should try to free some memory, so that the job can be executed