net.yacy.peers

Class Protocol

java.lang.Object

net.yacy.peers.Protocol

public final class Protocol
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Protocol.Post wrapper class for multi-post attempts to multiple IPs
private static class	Protocol.SearchResult
private static class	Protocol.WriteMetadataNodeToLocalIndexThread This thread is used to write a collection of URIMetadataNode documents to a segment allowing to be safely stopped.
private static class	Protocol.WriteToLocalIndexThread This thread is used to write a collection of Solr documents to a segment allowing to be safely stopped.

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.concurrent.atomic.AtomicInteger	metadataRetrievalRunning
private static CollectionSchema[]	snippetFields

Constructor Summary

Constructors

Constructor and Description
Protocol()

Method Summary

Methods

Modifier and Type	Method and Description
static boolean	authentifyRequest(serverObjects post, serverSwitch env)
static java.util.LinkedHashMap<java.lang.String,ContentBody>	basicRequestParts(Switchboard sb, java.lang.String targetHash, java.lang.String salt) put in all the essentials for routing and network authentication
protected static boolean	checkDocumentSize(SolrDocument doc, long maxSize) Only when maxSize is greater than zero, check that doc size is lower.
static java.util.Map<java.lang.String,java.lang.String>	crawlReceipt(Seed mySeed, Seed target, java.lang.String process, java.lang.String result, java.lang.String reason, URIMetadataNode entry, java.lang.String wordhashes)
static java.util.Map<java.lang.String,java.lang.String>	getProfile(Seed targetSeed)
static java.util.Map<java.lang.String,java.lang.String>	hello(Seed mySeed, PeerActions peerActions, java.lang.String targetAddress, java.lang.String targetHash) this is called to enrich the seed information by - own address (if peer is behind a nat/router) - check peer type (virgin/junior/senior/principal) to do this, we send a 'Hello' to another peer this carries the following information: 'iam' - own hash 'youare' - remote hash, to verify that we are correct 'key' - a session key that the remote peer may use to answer and the own seed string we expect the following information to be send back: - 'yourip' the ip of the connection peer (we) - 'yourtype' the type of this peer that the other peer checked by asking for a specific word and the remote seed string one exceptional failure case is when we know the other's peers hash, the other peers responds correctly but they appear to be another peer by comparisment of the other peer's hash this works of course only if we know the other peer's hash.
static ReferenceContainerCache<WebStructureGraph.HostReference>	loadIDXHosts(Seed target)
static java.util.Map<java.lang.String,java.lang.String>	permissionMessage(java.lang.String targetAddress, java.lang.String targetHash)
protected static int	primarySearch(SearchEvent event, java.lang.String wordhashes, java.lang.String excludehashes, java.lang.String language, Classification.ContentDomain contentdom, int count, long time, int maxDistance, int partitions, Seed target, SecondarySearchSuperviser secondarySearchSuperviser, Blacklist blacklist)
static RSSFeed	queryRemoteCrawlURLs(SeedDB seedDB, Seed target, int maxCount, long maxTime)
static long[]	queryRWICount(java.lang.String targetAddress, java.lang.String targetHash, int timeout)
static Seed	querySeed(Seed target, java.lang.String seedHash)
private static void	remoteSearchProcess(SearchEvent event, int count, long time, java.lang.String wordhashes, Seed target, Blacklist blacklist, Protocol.SearchResult result)
static java.lang.String	requestPartsToString(java.util.Map<java.lang.String,ContentBody> parts)
protected static int	secondarySearch(SearchEvent event, java.lang.String wordhashes, java.lang.String urlhashes, Classification.ContentDomain contentdom, int count, long time, int maxDistance, int partitions, Seed target, Blacklist blacklist)
protected static int	solrQuery(SearchEvent event, SolrQuery solrQuery, int offset, int count, Seed target, int partitions, Blacklist blacklist) Execute solr query against specified target.
static java.lang.String	transferIndex(SeedDB seeds, Seed targetSeed, ReferenceContainerCache<WordReference> indexes, HandleSet urlRefs, Segment segment, boolean gzipBody, int timeout) transfer the index.
private static java.util.Map<java.lang.String,java.lang.String>	transferRWI(Seed targetSeed, ReferenceContainerCache<WordReference> indexes, boolean gzipBody, int timeout)
private static java.util.Map<java.lang.String,java.lang.String>	transferURL(Seed targetSeed, java.lang.String[] uhs, HandleSet urlRefs, Segment segment, boolean gzipBody, int timeout)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

snippetFields

private static final CollectionSchema[] snippetFields

metadataRetrievalRunning

public static java.util.concurrent.atomic.AtomicInteger metadataRetrievalRunning

Constructor Detail

Protocol

public Protocol()

Method Detail

hello

public static java.util.Map<java.lang.String,java.lang.String> hello(Seed mySeed,
                                                     PeerActions peerActions,
                                                     java.lang.String targetAddress,
                                                     java.lang.String targetHash)

this is called to enrich the seed information by - own address (if peer is behind a nat/router) - check peer type (virgin/junior/senior/principal) to do this, we send a 'Hello' to another peer this carries the following information: 'iam' - own hash 'youare' - remote hash, to verify that we are correct 'key' - a session key that the remote peer may use to answer and the own seed string we expect the following information to be send back: - 'yourip' the ip of the connection peer (we) - 'yourtype' the type of this peer that the other peer checked by asking for a specific word and the remote seed string one exceptional failure case is when we know the other's peers hash, the other peers responds correctly but they appear to be another peer by comparisment of the other peer's hash this works of course only if we know the other peer's hash.

Returns:

the number of new seeds

querySeed

public static Seed querySeed(Seed target,
             java.lang.String seedHash)

queryRWICount

public static long[] queryRWICount(java.lang.String targetAddress,
                   java.lang.String targetHash,
                   int timeout)

queryRemoteCrawlURLs

public static RSSFeed queryRemoteCrawlURLs(SeedDB seedDB,
                           Seed target,
                           int maxCount,
                           long maxTime)

primarySearch

protected static int primarySearch(SearchEvent event,
                java.lang.String wordhashes,
                java.lang.String excludehashes,
                java.lang.String language,
                Classification.ContentDomain contentdom,
                int count,
                long time,
                int maxDistance,
                int partitions,
                Seed target,
                SecondarySearchSuperviser secondarySearchSuperviser,
                Blacklist blacklist)
                            throws java.lang.InterruptedException

Throws:

java.lang.InterruptedException

secondarySearch

protected static int secondarySearch(SearchEvent event,
                  java.lang.String wordhashes,
                  java.lang.String urlhashes,
                  Classification.ContentDomain contentdom,
                  int count,
                  long time,
                  int maxDistance,
                  int partitions,
                  Seed target,
                  Blacklist blacklist)
                              throws java.lang.InterruptedException

Throws:

java.lang.InterruptedException

remoteSearchProcess

private static void remoteSearchProcess(SearchEvent event,
                       int count,
                       long time,
                       java.lang.String wordhashes,
                       Seed target,
                       Blacklist blacklist,
                       Protocol.SearchResult result)
                                 throws SpaceExceededException,
                                        java.lang.InterruptedException

Throws:

SpaceExceededException

java.lang.InterruptedException

solrQuery

protected static int solrQuery(SearchEvent event,
            SolrQuery solrQuery,
            int offset,
            int count,
            Seed target,
            int partitions,
            Blacklist blacklist)
                        throws java.lang.InterruptedException

Execute solr query against specified target.

Parameters:

event - search event ot feed with results

solrQuery - solr query

offset - pagination start indice

count - expected maximum results

target - target peer to query. May be null : in that case, local peer is queried.

partitions -

blacklist - url list to exclude from results

Returns:

the size of results list

Throws:

java.lang.InterruptedException - when interrupt status on calling thread is detected while processing

checkDocumentSize

protected static boolean checkDocumentSize(SolrDocument doc,
                        long maxSize)

Only when maxSize is greater than zero, check that doc size is lower. To process in a reasonable amount of time, document size is not evaluated summing all fields sizes, but only against text_t field which is quite representative and might weigh some MB.

Parameters:

doc - document to verify. Must not be null.

maxSize - maximum allowed size in bytes

Returns:

true when document evaluated size is lower or equal than maxSize, or when maxSize is lower or equal than zero.

permissionMessage

public static java.util.Map<java.lang.String,java.lang.String> permissionMessage(java.lang.String targetAddress,
                                                                 java.lang.String targetHash)

crawlReceipt

public static java.util.Map<java.lang.String,java.lang.String> crawlReceipt(Seed mySeed,
                                                            Seed target,
                                                            java.lang.String process,
                                                            java.lang.String result,
                                                            java.lang.String reason,
                                                            URIMetadataNode entry,
                                                            java.lang.String wordhashes)

transferIndex

public static java.lang.String transferIndex(SeedDB seeds,
                             Seed targetSeed,
                             ReferenceContainerCache<WordReference> indexes,
                             HandleSet urlRefs,
                             Segment segment,
                             boolean gzipBody,
                             int timeout)

transfer the index. If the transmission fails, return a string describing the cause. If everything is ok, return null.

Parameters:

targetSeed -

indexes -

urlCache -

gzipBody -

timeout -

Returns:

transferRWI

private static java.util.Map<java.lang.String,java.lang.String> transferRWI(Seed targetSeed,
                                                           ReferenceContainerCache<WordReference> indexes,
                                                           boolean gzipBody,
                                                           int timeout)

transferURL

private static java.util.Map<java.lang.String,java.lang.String> transferURL(Seed targetSeed,
                                                           java.lang.String[] uhs,
                                                           HandleSet urlRefs,
                                                           Segment segment,
                                                           boolean gzipBody,
                                                           int timeout)

getProfile

public static java.util.Map<java.lang.String,java.lang.String> getProfile(Seed targetSeed)

loadIDXHosts

public static ReferenceContainerCache<WebStructureGraph.HostReference> loadIDXHosts(Seed target)

authentifyRequest

public static final boolean authentifyRequest(serverObjects post,
                        serverSwitch env)

basicRequestParts

public static final java.util.LinkedHashMap<java.lang.String,ContentBody> basicRequestParts(Switchboard sb,
                                                                      java.lang.String targetHash,
                                                                      java.lang.String salt)

put in all the essentials for routing and network authentication

Parameters:

sb -

targetHash -

salt -

Returns:

requestPartsToString

public static java.lang.String requestPartsToString(java.util.Map<java.lang.String,ContentBody> parts)