net.yacy.search.index

Class Segment

java.lang.Object

net.yacy.search.index.Segment

Direct Known Subclasses:

DocumentIndex

public class Segment
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	Segment.ReferenceReport A ReferenceReport object is a container for all references to a specific url.
class	Segment.ReferenceReportCache

Field Summary

Fields

Modifier and Type	Field and Description
static byte[]	catchallHash
static java.lang.String	catchallString
(package private) static Word	catchallWord
static java.lang.String	citationIndexName
static ReferenceFactory<CitationReference>	citationReferenceFactory
protected IndexTable	firstSeenIndex
static java.lang.String	firstseenIndexName
protected Fulltext	fulltext
private ConcurrentLog	log
static int	lowcachedivisor
protected IODispatcher	merger
private java.io.File	segmentPath
static long	targetFileSize
protected IndexCell<WordReference>	termIndex
static java.lang.String	termIndexName
protected IndexCell<CitationReference>	urlCitationIndex
static long	wCacheMaxAge
static int	wCacheMaxChunk
static ByteOrder	wordOrder
static ReferenceFactory<WordReference>	wordReferenceFactory
static int	writeBufferSize

Constructor Summary

Constructors

Constructor and Description
Segment(ConcurrentLog log, java.io.File segmentPath, java.io.File archivePath, CollectionConfiguration collectionConfiguration, WebgraphConfiguration webgraphConfiguration) create a new Segment

Method Summary

Methods

Modifier and Type	Method and Description
int	citationCount()
long	citationSegmentCount()
void	clear()
void	clearCaches()
void	close()
void	connectCitation(int entityCacheMaxSize, long maxFileSize)
boolean	connectedCitation()
boolean	connectedRWI()
void	connectRWI(int entityCacheMaxSize, long maxFileSize)
void	disconnectCitation()
void	disconnectRWI()
IndexTable	firstSeen()
Fulltext	fulltext()
long	getFirstSeenTime(byte[] urlhash)
long	getLoadTime(java.lang.String urlhash) get the load time of a resource.
java.io.File	getLocation()
Segment.ReferenceReportCache	getReferenceReportCache()
int	getWordCountGuess(java.lang.String word) get a guess about the word count.
void	putDocument(SolrInputDocument queueEntry) putDocument should not be called directly; instead, put queueEntries to indexingPutDocumentProcessor (this must be public, otherwise the WorkflowProcessor - calling by reflection - does not work) ATTENTION: do not remove! profiling tools will show that this is not called, which is not true (using reflection)
int	removeAllUrlReferences(byte[] urlhash, LoaderDispatcher loader, ClientIdentification.Agent agent, CacheStrategy cacheStrategy) find all the words in a specific resource and remove the url reference from every word index finally, delete the url entry
void	removeAllUrlReferences(HandleSet urls, LoaderDispatcher loader, ClientIdentification.Agent agent, CacheStrategy cacheStrategy)
int	RWIBufferCount()
long	RWICount()
long	RWISegmentCount()
void	setFirstSeenTime(byte[] urlhash, long time)
SolrInputDocument	storeDocument(DigestURL url, DigestURL referrerURL, java.util.Map<java.lang.String,java.util.regex.Pattern> collections, CrawlProfile crawlProfile, ResponseHeader responseHeader, Document document, Condenser condenser, SearchEvent searchEvent, java.lang.String sourceName, boolean storeToRWI, java.lang.String proxy, java.lang.String acceptLanguage)
void	storeRWI(byte[] termHash, WordReference entry)
void	storeRWI(ReferenceContainer<WordReference> wordContainer)
IndexCell<WordReference>	termIndex()
IndexCell<CitationReference>	urlCitation()
java.util.Iterator<DigestURL>	urlSelector(MultiProtocolURL stub, long maxtime, int maxcount) discover all urls that start with a given url stub
private static java.lang.String	votedLanguage(DigestURL url, java.lang.String urlNormalform, Document document, Condenser condenser)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

catchallString

public static final java.lang.String catchallString

See Also:

Constant Field Values

catchallHash

public static final byte[] catchallHash

catchallWord

static final Word catchallWord

wCacheMaxAge

public static final long wCacheMaxAge

See Also:

Constant Field Values

wCacheMaxChunk

public static final int wCacheMaxChunk

See Also:

Constant Field Values

lowcachedivisor

public static final int lowcachedivisor

See Also:

Constant Field Values

targetFileSize

public static final long targetFileSize

See Also:

Constant Field Values

writeBufferSize

public static final int writeBufferSize

See Also:

Constant Field Values

termIndexName

public static final java.lang.String termIndexName

See Also:

Constant Field Values

citationIndexName

public static final java.lang.String citationIndexName

See Also:

Constant Field Values

firstseenIndexName

public static final java.lang.String firstseenIndexName

See Also:

Constant Field Values

wordReferenceFactory

public static final ReferenceFactory<WordReference> wordReferenceFactory

citationReferenceFactory

public static final ReferenceFactory<CitationReference> citationReferenceFactory

wordOrder

public static final ByteOrder wordOrder

log

private final ConcurrentLog log

segmentPath

private final java.io.File segmentPath

fulltext

protected final Fulltext fulltext

termIndex

protected IndexCell<WordReference> termIndex

urlCitationIndex

protected IndexCell<CitationReference> urlCitationIndex

firstSeenIndex

protected IndexTable firstSeenIndex

merger

protected IODispatcher merger

Constructor Detail

Segment

public Segment(ConcurrentLog log,
       java.io.File segmentPath,
       java.io.File archivePath,
       CollectionConfiguration collectionConfiguration,
       WebgraphConfiguration webgraphConfiguration)
        throws java.io.IOException

create a new Segment

Parameters:

log -

segmentPath - that should be the path ponting to the directory "SEGMENT"

collectionSchema -

Throws:

java.io.IOException

Method Detail

connectedRWI

public boolean connectedRWI()

connectRWI

public void connectRWI(int entityCacheMaxSize,
              long maxFileSize)
                throws java.io.IOException

Throws:

java.io.IOException

disconnectRWI

public void disconnectRWI()

connectedCitation

public boolean connectedCitation()

connectCitation

public void connectCitation(int entityCacheMaxSize,
                   long maxFileSize)
                     throws java.io.IOException

Throws:

java.io.IOException

disconnectCitation

public void disconnectCitation()

citationCount

public int citationCount()

citationSegmentCount

public long citationSegmentCount()

fulltext

public Fulltext fulltext()

termIndex

public IndexCell<WordReference> termIndex()

urlCitation

public IndexCell<CitationReference> urlCitation()

firstSeen

public IndexTable firstSeen()

getReferenceReportCache

public Segment.ReferenceReportCache getReferenceReportCache()

RWICount

public long RWICount()

RWISegmentCount

public long RWISegmentCount()

RWIBufferCount

public int RWIBufferCount()

getWordCountGuess

public int getWordCountGuess(java.lang.String word)

get a guess about the word count. This is only a guess because it uses the term index if present and this index may be influenced by index transmission processes in its statistic word distribution. However, it can be a hint for heuristics which use the word count. Please do NOT use this if the termIndex is not present because it otherwise uses the solr index which makes it painfully slow.

Parameters:

word -

Returns:

the number of references for this word.

setFirstSeenTime

public void setFirstSeenTime(byte[] urlhash,
                    long time)

getFirstSeenTime

public long getFirstSeenTime(byte[] urlhash)

getLoadTime

public long getLoadTime(java.lang.String urlhash)
                 throws java.io.IOException

get the load time of a resource.

Parameters:

urlHash -

Returns:

the time in milliseconds since epoch for the load time or -1 if the document does not exist

Throws:

java.io.IOException

urlSelector

public java.util.Iterator<DigestURL> urlSelector(MultiProtocolURL stub,
                                        long maxtime,
                                        int maxcount)

discover all urls that start with a given url stub

Parameters:

stub -

Returns:

an iterator for all matching urls

clear

public void clear()

clearCaches

public void clearCaches()

getLocation

public java.io.File getLocation()

close

public void close()

votedLanguage

private static java.lang.String votedLanguage(DigestURL url,
                             java.lang.String urlNormalform,
                             Document document,
                             Condenser condenser)

storeRWI

public void storeRWI(ReferenceContainer<WordReference> wordContainer)
              throws java.io.IOException,
                     SpaceExceededException

Throws:

java.io.IOException

SpaceExceededException

storeRWI

public void storeRWI(byte[] termHash,
            WordReference entry)
              throws java.io.IOException,
                     SpaceExceededException

Throws:

java.io.IOException

SpaceExceededException

putDocument

public void putDocument(SolrInputDocument queueEntry)

putDocument should not be called directly; instead, put queueEntries to indexingPutDocumentProcessor (this must be public, otherwise the WorkflowProcessor - calling by reflection - does not work) ATTENTION: do not remove! profiling tools will show that this is not called, which is not true (using reflection)

Parameters:

queueEntry -

Throws:

java.io.IOException

storeDocument

public SolrInputDocument storeDocument(DigestURL url,
                              DigestURL referrerURL,
                              java.util.Map<java.lang.String,java.util.regex.Pattern> collections,
                              CrawlProfile crawlProfile,
                              ResponseHeader responseHeader,
                              Document document,
                              Condenser condenser,
                              SearchEvent searchEvent,
                              java.lang.String sourceName,
                              boolean storeToRWI,
                              java.lang.String proxy,
                              java.lang.String acceptLanguage)

removeAllUrlReferences

public void removeAllUrlReferences(HandleSet urls,
                          LoaderDispatcher loader,
                          ClientIdentification.Agent agent,
                          CacheStrategy cacheStrategy)

removeAllUrlReferences

public int removeAllUrlReferences(byte[] urlhash,
                         LoaderDispatcher loader,
                         ClientIdentification.Agent agent,
                         CacheStrategy cacheStrategy)

find all the words in a specific resource and remove the url reference from every word index finally, delete the url entry

Parameters:

urlhash - the hash of the url that shall be removed

loader -

cacheStrategy -

Returns:

number of removed words