net.yacy.kelondro.rwi

Class IndexCell<ReferenceType extends Reference>

java.lang.Object

net.yacy.kelondro.rwi.AbstractIndex<ReferenceType>

net.yacy.kelondro.rwi.AbstractBufferedIndex<ReferenceType>

net.yacy.kelondro.rwi.IndexCell<ReferenceType>

All Implemented Interfaces:

java.lang.Iterable<ReferenceContainer<ReferenceType>>, BufferedIndex<ReferenceType>, Index<ReferenceType>

public final class IndexCell<ReferenceType extends Reference>
extends AbstractBufferedIndex<ReferenceType>
implements BufferedIndex<ReferenceType>, java.lang.Iterable<ReferenceContainer<ReferenceType>>

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	IndexCell.FlushThread
private static class	IndexCell.RemoveReducer<ReferenceType extends Reference>

Field Summary

Fields

Modifier and Type	Field and Description
private ReferenceContainerArray<ReferenceType>	array
private static long	cleanupCycle
private ComparableARC<byte[],java.lang.Integer>	countCache
private static long	dumpCycle
private boolean	flushShallRun
private java.lang.Thread	flushThread
private long	lastCleanup
private long	lastDump
private long	maxFileSize
private int	maxRamEntries
private IODispatcher	merger
private ReferenceContainerCache<ReferenceType>	ram
private java.util.Map<byte[],HandleSet>	removeDelayedURLs
private long	targetFileSize
private int	writeBufferSize

Fields inherited from class net.yacy.kelondro.rwi.AbstractIndex

factory

Constructor Summary

Constructors

Constructor and Description
IndexCell(java.io.File cellPath, java.lang.String prefix, ReferenceFactory<ReferenceType> factory, ByteOrder termOrder, int termSize, int maxRamEntries, long targetFileSize, long maxFileSize, int writeBufferSize, IODispatcher merger)

Method Summary

Methods

Modifier and Type	Method and Description
void	add(byte[] termHash, ReferenceType entry) add a single reference to the reverse index if no references to the word are stored, the a new entry is added, if there are already references to the word hash stored, then the old and the new references are merged
void	add(ReferenceContainer<ReferenceType> newEntries) add entries to the cell: this adds the new entries always to the RAM part, never to BLOBs
void	clear() clear the RAM and BLOB part, deletes everything in the cell
void	clearCache()
void	close() when a cell is closed, the current RAM is dumped to a file which will be opened as BLOB file the next time a cell is opened.
int	count(byte[] termHash) count number of references for a given term this method may cause strong IO load if called too frequently.
void	delete(byte[] termHash) delete all references for a word the difference to 'remove' is, that the removed element is not returned
int	deleteOld(int minsize, long maxtime)
ReferenceContainer<ReferenceType>	get(byte[] termHash, HandleSet urlselection) all containers in the BLOBs and the RAM are merged and returned.
long	getBufferMaxAge() return the date of the most recent buffer entry
int	getBufferMaxReferences() return the maximum number of references, that one buffer entry has stored
long	getBufferMinAge() return the date of the oldest buffer entry
int	getBufferSize() get the size of the buffer content
long	getBufferSizeBytes() calculate the memory that is taken by the buffer.
int	getSegmentCount()
boolean	has(byte[] termHash) checks if there is any container for this termHash, either in RAM or any BLOB
boolean	isEmpty()
java.util.Iterator<ReferenceContainer<ReferenceType>>	iterator()
private java.util.Collection<byte[]>	keys4LargeReferences(int minsize, long maxtime)
int	minMem() calculate needed memory
CloneableIterator<ReferenceContainer<ReferenceType>>	referenceContainerIterator(byte[] startTermHash, boolean rot, boolean excludePrivate) iterate all references from the beginning of a specific word hash
CloneableIterator<ReferenceContainer<ReferenceType>>	referenceContainerIterator(byte[] startTermHash, boolean rot, boolean excludePrivate, boolean ram) iterate over entries in index.
CloneableIterator<Rating<byte[]>>	referenceCountIterator(byte[] starttermHash, boolean rot, boolean excludePrivate) iterate all references from the beginning of a specific word hash
ReferenceContainer<ReferenceType>	remove(byte[] termHash) deleting a container affects the containers in RAM and all the BLOB files the deleted containers are merged and returned as result of the method
boolean	remove(byte[] termHash, byte[] urlHashBytes) remove a specific reference entry
int	remove(byte[] termHash, HandleSet urlHashes) remove url references from a selected word hash.
void	removeDelayed()
void	removeDelayed(byte[] termHash, byte[] urlHashBytes)
void	setBufferMaxWordCount(int maxWords) set the size of the buffer, which can be defined with a given maximum number of words that shall be stored.
private boolean	shrink(long targetFileSize, long maxFileSize)
int	size() the number of all references
private int[]	sizes()
int	sizesMax()
int	termKeyLength() every index entry is made for a term which has a fixed size
ByteOrder	termKeyOrdering() return the order that is used for the storage of the word hashes

Methods inherited from class net.yacy.kelondro.rwi.AbstractBufferedIndex

referenceContainer

Methods inherited from class net.yacy.kelondro.rwi.AbstractIndex

merge, query, referenceRow, remove, searchConjunction

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.yacy.kelondro.rwi.BufferedIndex

referenceContainer

Methods inherited from interface net.yacy.kelondro.rwi.Index

merge, referenceRow, remove, searchConjunction

Field Detail

cleanupCycle

private static final long cleanupCycle

See Also:

Constant Field Values

dumpCycle

private static final long dumpCycle

See Also:

Constant Field Values

array

private final ReferenceContainerArray<ReferenceType extends Reference> array

ram

private ReferenceContainerCache<ReferenceType extends Reference> ram

countCache

private final ComparableARC<byte[],java.lang.Integer> countCache

maxRamEntries

private int maxRamEntries

merger

private IODispatcher merger

lastCleanup

private long lastCleanup

lastDump

private long lastDump

targetFileSize

private final long targetFileSize

maxFileSize

private final long maxFileSize

writeBufferSize

private final int writeBufferSize

removeDelayedURLs

private final java.util.Map<byte[],HandleSet> removeDelayedURLs

flushShallRun

private boolean flushShallRun

flushThread

private final java.lang.Thread flushThread

Constructor Detail

IndexCell

public IndexCell(java.io.File cellPath,
         java.lang.String prefix,
         ReferenceFactory<ReferenceType> factory,
         ByteOrder termOrder,
         int termSize,
         int maxRamEntries,
         long targetFileSize,
         long maxFileSize,
         int writeBufferSize,
         IODispatcher merger)
          throws java.io.IOException

Throws:

java.io.IOException

Method Detail

shrink

private boolean shrink(long targetFileSize,
             long maxFileSize)

deleteOld

public int deleteOld(int minsize,
            long maxtime)
              throws java.io.IOException

Throws:

java.io.IOException

keys4LargeReferences

private java.util.Collection<byte[]> keys4LargeReferences(int minsize,
                                                long maxtime)
                                                   throws java.io.IOException

Throws:

java.io.IOException

termKeyLength

public int termKeyLength()

every index entry is made for a term which has a fixed size

Specified by:

termKeyLength in interface Index<ReferenceType extends Reference>

Returns:

the size of the term

add

public void add(ReferenceContainer<ReferenceType> newEntries)
         throws java.io.IOException,
                SpaceExceededException

add entries to the cell: this adds the new entries always to the RAM part, never to BLOBs

Specified by:

add in interface Index<ReferenceType extends Reference>

Parameters:

newEntries - the References to be merged with existing references

Throws:

java.io.IOException

SpaceExceededException

add

public void add(byte[] termHash,
       ReferenceType entry)
         throws java.io.IOException,
                SpaceExceededException

Description copied from interface: Index

add a single reference to the reverse index if no references to the word are stored, the a new entry is added, if there are already references to the word hash stored, then the old and the new references are merged

Specified by:

add in interface Index<ReferenceType extends Reference>

Throws:

java.io.IOException

SpaceExceededException

has

public boolean has(byte[] termHash)

checks if there is any container for this termHash, either in RAM or any BLOB

Specified by:

has in interface Index<ReferenceType extends Reference>

Returns:

true if references exist, false if not

count

public int count(byte[] termHash)

count number of references for a given term this method may cause strong IO load if called too frequently.

Specified by:

count in interface Index<ReferenceType extends Reference>

Returns:

the number of references to the given word

get

public ReferenceContainer<ReferenceType> get(byte[] termHash,
                                    HandleSet urlselection)
                                                        throws java.io.IOException

all containers in the BLOBs and the RAM are merged and returned. Please be aware that the returned values may be top-level cloned ReferenceContainers or direct links to containers If the containers are modified after they are returned, they MAY alter the stored index.

Specified by:

get in interface Index<ReferenceType extends Reference>

Returns:

a container with merged ReferenceContainer from RAM and the file array or null if there is no data to be returned

Throws:

java.io.IOException

remove

public ReferenceContainer<ReferenceType> remove(byte[] termHash)
                                                           throws java.io.IOException

deleting a container affects the containers in RAM and all the BLOB files the deleted containers are merged and returned as result of the method

Specified by:

remove in interface Index<ReferenceType extends Reference>

Returns:

the deleted references

Throws:

java.io.IOException

delete

public void delete(byte[] termHash)
            throws java.io.IOException

Description copied from interface: Index

delete all references for a word the difference to 'remove' is, that the removed element is not returned

Specified by:

delete in interface Index<ReferenceType extends Reference>

Throws:

java.io.IOException

removeDelayed

public void removeDelayed(byte[] termHash,
                 byte[] urlHashBytes)

Specified by:

removeDelayed in interface Index<ReferenceType extends Reference>

removeDelayed

public void removeDelayed()
                   throws java.io.IOException

Specified by:

removeDelayed in interface Index<ReferenceType extends Reference>

Throws:

java.io.IOException

remove

public int remove(byte[] termHash,
         HandleSet urlHashes)
           throws java.io.IOException

remove url references from a selected word hash. this deletes also in the BLOB files, which means that there exists new gap entries after the deletion The gaps are never merged in place, but can be eliminated when BLOBs are merged into new BLOBs. This returns the sum of all url references that have been removed

Specified by:

remove in interface Index<ReferenceType extends Reference>

Parameters:

termHash - the key for the references

Returns:

Throws:

java.io.IOException

remove

public boolean remove(byte[] termHash,
             byte[] urlHashBytes)
               throws java.io.IOException

Description copied from interface: Index

remove a specific reference entry

Specified by:

remove in interface Index<ReferenceType extends Reference>

urlHashBytes - the key for the reference entry to be removed

Returns:

Throws:

java.io.IOException

iterator

public java.util.Iterator<ReferenceContainer<ReferenceType>> iterator()

Specified by:

iterator in interface java.lang.Iterable<ReferenceContainer<ReferenceType extends Reference>>

referenceCountIterator

public CloneableIterator<Rating<byte[]>> referenceCountIterator(byte[] starttermHash,
                                                       boolean rot,
                                                       boolean excludePrivate)

Description copied from interface: Index

iterate all references from the beginning of a specific word hash

Specified by:

referenceCountIterator in interface Index<ReferenceType extends Reference>

rot - if true, then rotate at the end to the beginning

Returns:

referenceContainerIterator

public CloneableIterator<ReferenceContainer<ReferenceType>> referenceContainerIterator(byte[] startTermHash,
                                                                              boolean rot,
                                                                              boolean excludePrivate)

Description copied from interface: Index

iterate all references from the beginning of a specific word hash

Specified by:

referenceContainerIterator in interface Index<ReferenceType extends Reference>

rot - if true, then rotate at the end to the beginning

Returns:

referenceContainerIterator

public CloneableIterator<ReferenceContainer<ReferenceType>> referenceContainerIterator(byte[] startTermHash,
                                                                              boolean rot,
                                                                              boolean excludePrivate,
                                                                              boolean ram)

Description copied from interface: BufferedIndex

iterate over entries in index. this method differs from the iterator in an Index object in such a way that it has the additional 'buffer' flag. When using this method, the iteration goes only over the buffer content, or over the backend-content, but not over a merged content.

Specified by:

referenceContainerIterator in interface BufferedIndex<ReferenceType extends Reference>

Returns:

clear

public void clear()
           throws java.io.IOException

clear the RAM and BLOB part, deletes everything in the cell

Specified by:

clear in interface Index<ReferenceType extends Reference>

Throws:

java.io.IOException

clearCache

public void clearCache()

close

public void close()

when a cell is closed, the current RAM is dumped to a file which will be opened as BLOB file the next time a cell is opened. A name for the dump is automatically generated and is composed of the current date and the cell salt

Specified by:

close in interface Index<ReferenceType extends Reference>

isEmpty

public boolean isEmpty()

size

public int size()

Description copied from interface: Index

the number of all references

Specified by:

size in interface Index<ReferenceType extends Reference>

Returns:

the nnumber of all references

sizes

private int[] sizes()

sizesMax

public int sizesMax()

getSegmentCount

public int getSegmentCount()

minMem

public int minMem()

Description copied from interface: Index

calculate needed memory

Specified by:

minMem in interface Index<ReferenceType extends Reference>

Returns:

the memory needed to operate the object

termKeyOrdering

public ByteOrder termKeyOrdering()

Description copied from interface: Index

return the order that is used for the storage of the word hashes

Specified by:

termKeyOrdering in interface Index<ReferenceType extends Reference>

Returns:

getBufferMaxAge

public long getBufferMaxAge()

Description copied from interface: BufferedIndex

return the date of the most recent buffer entry

Specified by:

getBufferMaxAge in interface BufferedIndex<ReferenceType extends Reference>

Returns:

a time as milliseconds from epoch

getBufferMaxReferences

public int getBufferMaxReferences()

Description copied from interface: BufferedIndex

return the maximum number of references, that one buffer entry has stored

Specified by:

getBufferMaxReferences in interface BufferedIndex<ReferenceType extends Reference>

Returns:

getBufferMinAge

public long getBufferMinAge()

Description copied from interface: BufferedIndex

return the date of the oldest buffer entry

Specified by:

getBufferMinAge in interface BufferedIndex<ReferenceType extends Reference>

Returns:

a time as milliseconds from epoch

getBufferSize

public int getBufferSize()

Description copied from interface: BufferedIndex

get the size of the buffer content

Specified by:

getBufferSize in interface BufferedIndex<ReferenceType extends Reference>

Returns:

number of word references

getBufferSizeBytes

public long getBufferSizeBytes()

Description copied from interface: BufferedIndex

calculate the memory that is taken by the buffer. This does not simply return a variable content. it is necessary to iterate over all entries to get the whole size. please use this method with great care

Specified by:

getBufferSizeBytes in interface BufferedIndex<ReferenceType extends Reference>

Returns:

number of bytes that the buffer has allocated

setBufferMaxWordCount

public void setBufferMaxWordCount(int maxWords)

Description copied from interface: BufferedIndex

set the size of the buffer, which can be defined with a given maximum number of words that shall be stored. Because an arbitrary number of references can be stored at each word entry, this does not limit the memory that the buffer takes. It is possible to monitor the memory occupation of the buffer with the getBufferSizeBytes() method, and then adopt the buffer size with a new word number limit.

Specified by:

setBufferMaxWordCount in interface BufferedIndex<ReferenceType extends Reference>