net.yacy.cora.federate.solr.connector

Class AbstractSolrConnector

java.lang.Object

net.yacy.cora.federate.solr.connector.AbstractSolrConnector

All Implemented Interfaces:

java.lang.Iterable<java.lang.String>, SolrConnector

Direct Known Subclasses:

MirrorSolrConnector, SolrServerConnector

public abstract class AbstractSolrConnector
extends java.lang.Object
implements SolrConnector

Nested Class Summary

Nested classes/interfaces inherited from interface net.yacy.cora.federate.solr.connector.SolrConnector

SolrConnector.LoadTimeURL

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CATCHALL_DTERM
static java.lang.String	CATCHALL_QUERY
static java.lang.String	CATCHALL_TERM
static SolrQuery	catchallQuery
static SolrQuery	catchSuccessQuery
protected static int	pagesize_docs
protected static int	pagesize_ids
static SolrDocument	POISON_DOCUMENT
static java.lang.String	POISON_ID
protected static java.util.Set<java.lang.String>	SOLR_ID_and_LOAD_DATE_FIELDS
protected static java.util.Set<java.lang.String>	SOLR_ID_FIELDS

Constructor Summary

Constructors

Constructor and Description
AbstractSolrConnector()

Method Summary

Methods

Modifier and Type	Method and Description
java.util.concurrent.BlockingQueue<SolrDocument>	concurrentDocumentsByQueries(java.util.List<java.lang.String> querystrings, java.lang.String sort, int offset, int maxcount, long maxtime, int buffersize, int concurrency, boolean prefetchIDs, java.lang.String... fields) Get results from solr queries as a stream of documents.
private java.util.concurrent.BlockingQueue<SolrDocument>	concurrentDocumentsByQueriesNoPrefetch(java.util.List<java.lang.String> querystrings, java.lang.String sort, int offset, int maxcount, long maxtime, int buffersize, int concurrency, java.lang.String... fields)
java.util.concurrent.BlockingQueue<SolrDocument>	concurrentDocumentsByQuery(java.lang.String querystring, java.lang.String sort, int offset, int maxcount, long maxtime, int buffersize, int concurrency, boolean prefetchIDs, java.lang.String... fields) Get results from a solr query as a stream of documents.
java.util.concurrent.BlockingQueue<java.lang.String>	concurrentIDsByQueries(java.util.List<java.lang.String> querystrings, java.lang.String sort, int offset, int maxcount, long maxtime, int buffersize, int concurrency) get a document id result stream from a set of solr queries.
java.util.concurrent.BlockingQueue<java.lang.String>	concurrentIDsByQuery(java.lang.String querystring, java.lang.String sort, int offset, int maxcount, long maxtime, int buffersize, int concurrency) get a document id result stream from a solr query.
protected static java.lang.String[]	ensureEssentialFieldsIncluded(java.lang.String[] fields) check if fields contain id and load_date_dt date
long	getCountByQuery(java.lang.String querystring) get the number of results when this query is done.
SolrDocument	getDocumentById(java.lang.String id, java.lang.String... fields) get a document from solr by given key for the id-field
SolrDocumentList	getDocumentListByQuery(java.lang.String querystring, java.lang.String sort, int offset, int count, java.lang.String... fields) get a query result from solr to get all results set the query String to "*:*"
java.util.LinkedHashMap<java.lang.String,ReversibleScoreMap<java.lang.String>>	getFacets(java.lang.String query, int maxresults, java.lang.String... fields) get facets of the index: a list of lists with values that are most common in a specific field
protected static SolrConnector.LoadTimeURL	getLoadTimeURL(java.lang.Object doc)
SolrConnector.LoadTimeURL	getLoadTimeURL(java.lang.String id) check if a given document, identified by url hash as document id exists
static SolrQuery	getSolrQuery(java.lang.String querystring, java.lang.String sort, int offset, int count, java.lang.String... fields)
java.util.Iterator<java.lang.String>	iterator()
private SolrInputDocument	partialUpdatePatch(SolrInputDocument docIn)
void	update(java.util.Collection<SolrInputDocument> solrdoc) Update a collection of solr input documents.
void	update(SolrInputDocument solrdoc) Update a solr document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.yacy.cora.federate.solr.connector.SolrConnector

add, add, bufferSize, clear, clearCaches, close, commit, deleteById, deleteByIds, deleteByQuery, getDocumentListByParams, getResponseByParams, getSegmentCount, getSize, isClosed, optimize

Field Detail

SOLR_ID_FIELDS

protected static java.util.Set<java.lang.String> SOLR_ID_FIELDS

SOLR_ID_and_LOAD_DATE_FIELDS

protected static java.util.Set<java.lang.String> SOLR_ID_and_LOAD_DATE_FIELDS

POISON_DOCUMENT

public static final SolrDocument POISON_DOCUMENT

POISON_ID

public static final java.lang.String POISON_ID

See Also:

Constant Field Values

CATCHALL_TERM

public static final java.lang.String CATCHALL_TERM

See Also:

Constant Field Values

CATCHALL_DTERM

public static final java.lang.String CATCHALL_DTERM

See Also:

Constant Field Values

CATCHALL_QUERY

public static final java.lang.String CATCHALL_QUERY

See Also:

Constant Field Values

catchallQuery

public static final SolrQuery catchallQuery

catchSuccessQuery

public static final SolrQuery catchSuccessQuery

pagesize_docs

protected static final int pagesize_docs

See Also:

Constant Field Values

pagesize_ids

protected static final int pagesize_ids

See Also:

Constant Field Values

Constructor Detail

AbstractSolrConnector

public AbstractSolrConnector()

Method Detail

getLoadTimeURL

protected static SolrConnector.LoadTimeURL getLoadTimeURL(java.lang.Object doc)

ensureEssentialFieldsIncluded

protected static java.lang.String[] ensureEssentialFieldsIncluded(java.lang.String[] fields)

check if fields contain id and load_date_dt date

Parameters:

fields -

Returns:

fields with added id and load_date_dt if necessary

concurrentDocumentsByQuery

public java.util.concurrent.BlockingQueue<SolrDocument> concurrentDocumentsByQuery(java.lang.String querystring,
                                                                          java.lang.String sort,
                                                                          int offset,
                                                                          int maxcount,
                                                                          long maxtime,
                                                                          int buffersize,
                                                                          int concurrency,
                                                                          boolean prefetchIDs,
                                                                          java.lang.String... fields)

Get results from a solr query as a stream of documents. The result queue is considered as terminated if AbstractSolrConnector.POISON_DOCUMENT is returned. The method returns immediately and feeds the search results into the queue

Specified by:

concurrentDocumentsByQuery in interface SolrConnector

Parameters:

querystring - the solr query string

sort - the solr sort string, may be null to be not used

offset - first result offset

maxcount - the maximum number of results

maxtime - the maximum time in milliseconds

buffersize - the size of an ArrayBlockingQueue; if <= 0 then a LinkedBlockingQueue is used

concurrency - is the number of AbstractSolrConnector.POISON_DOCUMENT entries to add at the end of the feed

prefetchIDs - if true, then first all IDs are fetched and then all documents are queries by the ID. If false then documents are retrieved directly

fields - list of fields

Returns:

a blocking queue which is terminated with AbstractSolrConnector.POISON_DOCUMENT as last element

concurrentDocumentsByQueries

public java.util.concurrent.BlockingQueue<SolrDocument> concurrentDocumentsByQueries(java.util.List<java.lang.String> querystrings,
                                                                            java.lang.String sort,
                                                                            int offset,
                                                                            int maxcount,
                                                                            long maxtime,
                                                                            int buffersize,
                                                                            int concurrency,
                                                                            boolean prefetchIDs,
                                                                            java.lang.String... fields)

Get results from solr queries as a stream of documents. The result queue is considered as terminated if AbstractSolrConnector.POISON_DOCUMENT is returned. The method returns immediately and feeds the search results into the queue

Specified by:

concurrentDocumentsByQueries in interface SolrConnector

Parameters:

querystrings - the list of solr query strings

sort - the solr sort string, may be null to be not used

offset - first result offset

maxcount - the maximum number of results

maxtime - the maximum time in milliseconds

buffersize - the size of an ArrayBlockingQueue; if <= 0 then a LinkedBlockingQueue is used

concurrency - is the number of AbstractSolrConnector.POISON_DOCUMENT entries to add at the end of the feed

prefetchIDs - if true, then first all IDs are fetched and then all documents are queries by the ID. If false then documents are retrieved directly

fields - list of fields

Returns:

a blocking queue which is terminated with AbstractSolrConnector.POISON_DOCUMENT as last element

concurrentDocumentsByQueriesNoPrefetch

private java.util.concurrent.BlockingQueue<SolrDocument> concurrentDocumentsByQueriesNoPrefetch(java.util.List<java.lang.String> querystrings,
                                                                                      java.lang.String sort,
                                                                                      int offset,
                                                                                      int maxcount,
                                                                                      long maxtime,
                                                                                      int buffersize,
                                                                                      int concurrency,
                                                                                      java.lang.String... fields)

concurrentIDsByQuery

public java.util.concurrent.BlockingQueue<java.lang.String> concurrentIDsByQuery(java.lang.String querystring,
                                                                        java.lang.String sort,
                                                                        int offset,
                                                                        int maxcount,
                                                                        long maxtime,
                                                                        int buffersize,
                                                                        int concurrency)

get a document id result stream from a solr query. The result queue is considered as terminated if AbstractSolrConnector.POISON_ID is returned. The method returns immediately and feeds the search results into the queue

Specified by:

concurrentIDsByQuery in interface SolrConnector

Parameters:

querystring -

sort - the solr sort string, may be null to be not used

offset -

maxcount -

buffersize - the size of an ArrayBlockingQueue; if <= 0 then a LinkedBlockingQueue is used

concurrency - is the number of AbstractSolrConnector.POISON_ID entries to add at the end of the feed

Returns:

a list of ids in q blocking queue which is terminated with a number of AbstractSolrConnector.POISON_ID

concurrentIDsByQueries

public java.util.concurrent.BlockingQueue<java.lang.String> concurrentIDsByQueries(java.util.List<java.lang.String> querystrings,
                                                                          java.lang.String sort,
                                                                          int offset,
                                                                          int maxcount,
                                                                          long maxtime,
                                                                          int buffersize,
                                                                          int concurrency)

get a document id result stream from a set of solr queries. The result queue is considered as terminated if AbstractSolrConnector.POISON_ID is returned. The method returns immediately and feeds the search results into the queue

Specified by:

concurrentIDsByQueries in interface SolrConnector

Parameters:

querystring - a list of query strings

sort - the solr sort string, may be null to be not used

offset - common offset of all queries

maxcount - maximum count for each query

buffersize - the size of an ArrayBlockingQueue; if <= 0 then a LinkedBlockingQueue is used

concurrency - is the number of AbstractSolrConnector.POISON_ID entries to add at the end of the feed

Returns:

a list of ids in q blocking queue which is terminated with a number of AbstractSolrConnector.POISON_ID

iterator

public java.util.Iterator<java.lang.String> iterator()

Specified by:

iterator in interface java.lang.Iterable<java.lang.String>

getDocumentListByQuery

public SolrDocumentList getDocumentListByQuery(java.lang.String querystring,
                                      java.lang.String sort,
                                      int offset,
                                      int count,
                                      java.lang.String... fields)
                                        throws java.io.IOException

get a query result from solr to get all results set the query String to "*:*"

Specified by:

getDocumentListByQuery in interface SolrConnector

Parameters:

querystring -

sort - the solr sort string, may be null to be not used

offset - the first result offset

count - number of wanted results

fields - list of fields

Throws:

java.io.IOException

getSolrQuery

public static SolrQuery getSolrQuery(java.lang.String querystring,
                     java.lang.String sort,
                     int offset,
                     int count,
                     java.lang.String... fields)

getLoadTimeURL

public SolrConnector.LoadTimeURL getLoadTimeURL(java.lang.String id)
                                         throws java.io.IOException

check if a given document, identified by url hash as document id exists

Specified by:

getLoadTimeURL in interface SolrConnector

Parameters:

id - the url hash and document id

Returns:

metadata if any entry in solr exists, null otherwise

Throws:

java.io.IOException

getCountByQuery

public long getCountByQuery(java.lang.String querystring)
                     throws java.io.IOException

get the number of results when this query is done. This should only be called if the actual result is never used, and only the count is interesting

Specified by:

getCountByQuery in interface SolrConnector

Parameters:

querystring -

Returns:

the number of results for this query

Throws:

java.io.IOException

getFacets

public java.util.LinkedHashMap<java.lang.String,ReversibleScoreMap<java.lang.String>> getFacets(java.lang.String query,
                                                                                       int maxresults,
                                                                                       java.lang.String... fields)
                                                                                         throws java.io.IOException

get facets of the index: a list of lists with values that are most common in a specific field

Specified by:

getFacets in interface SolrConnector

Parameters:

query - a query which is performed to get the facets

fields - the field names which are selected as facet

maxresults - the maximum size of the resulting maps

Returns:

a map with key = facet field name, value = an ordered map of field values for that field

Throws:

java.io.IOException

getDocumentById

public SolrDocument getDocumentById(java.lang.String id,
                           java.lang.String... fields)
                             throws java.io.IOException

Description copied from interface: SolrConnector

get a document from solr by given key for the id-field

Specified by:

getDocumentById in interface SolrConnector

fields - list of fields

Returns:

one result or null if no result exists

Throws:

java.io.IOException

update

public void update(SolrInputDocument solrdoc)
            throws java.io.IOException,
                   SolrException

Update a solr document. This will write only a partial update for all fields given in the SolrInputDocument and leaves all other fields untouched.

Specified by:

update in interface SolrConnector

Parameters:

solrdoc -

Throws:

java.io.IOException

SolrException

update

public void update(java.util.Collection<SolrInputDocument> solrdoc)
            throws java.io.IOException,
                   SolrException

Update a collection of solr input documents. This will write only a partial update for all fields given in the SolrInputDocuments and leaves all other fields untouched.

Specified by:

update in interface SolrConnector

Parameters:

solrdocs -

Throws:

java.io.IOException

SolrException

partialUpdatePatch

private SolrInputDocument partialUpdatePatch(SolrInputDocument docIn)