YaCy Release 1.68

removed jena library and all code that depended on jena. When jena was
introduced, it was also used for search facets. The generic search
facets are now deduced from generic solr fields which makes jena as tool
for facet semantics superfluous.

prepare for a language navigator
works fine to restrict language for local solrSearches.
More work needs to be done to make rwi/remote searches respect the modifier.language restriction.

bump to httpclient-4.3.2

configurable max-load values for YaCy-Threads:
try lower values on smal systems like a Pi 

- removed old metadata database and all migration code
- refactored all code which uses URIMetadataRow as standard for word
hash length and word hash ordering and moved that to the class 'Word',
becuase the class URIMetadataRow defined the old metadata data structure
and should be superfluous in the future
- removed unused methods from URIMetadataRow as preparation for further
removal of that class

adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString
- since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic,
adjusted ConfigHeuristic to use OpensearchHeuristic settings only.
For this the default OSD search target list is made available (copied) by default and the other configs are removed.

- the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object,
but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns
just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers)

- started to adjust internal html href references from absolute to relative (currently it is mixed).
For future development we should prefer relative href targets (less trouble with context aware  servlets)

do YaCy p2p connections using a timeout-request which covers the http
request into a separate thread and ignores the furthure result of a
request if that does not answer within the requested time-out. This is a
try to solve a problem with the peer-ping, which hangs whenever a peer
appears to be dead or blocked.

implemented DIGEST authentication, which is for remote login more secure
as BASIC were pwd is transmitted near clear text (B64enc).
This has some implication as RFC 2617 requires and recommends a password hash MD5(user:realm:pwd) for DIGEST.

!!! before activating DIGEST you have to reassign all passwords !!! to allow new calculation of the hash
- default authentication is still BASIC
- configuration at this time only manually in (DATA/settings) or  defaults/web.xml  (<auth-method>
- the realmname is in defaults/yacy.init  adminRealm=YaCy-AdminUI
- fyi: the realmname is shown on login screen
- changing the realm name invalidates all passwords - but for security you are encouraged to do so (as localhostadmin)
- implemented to support both, old hashes for BASIC and new hashes for BASIC and DIGEST
- to differentiate old / new hash the in Jetty used hash-prefix "MD5:" is used for new pwd-hashes (  "MD5:hash" )

remote search peer selection schema change:
- all non-dht targets (previously separated into 'robinson' for dht-like
queries and 'node' for solr queries) are non 'extra' peers, which are
queries using solr
- these extra-peers are now selected using a ranking on last-seen,
peer-tag-matches, node-peer flags, peer age, and link count. The ranking
is done using a weight and a random factor.
- the number of extra peers is 50% of the dht peers
- the dht peers now exclude too young peers to prevent bad results
during strong growth of the network
- the number of dht peers (and therefore extra-peers) is reduced when
the memory of the peer is low and/or some documents still appear in the
indexing-queue. This shall prevent a peer from deadlocks when p2p
queries are made in a fast sequence on weak hardware.

making root context (htroot) a WebAppContext
- this allows additional features, like servlet configuration via web.xml and many more things.
- currently the standard servlets are still configured in the code (so the supplied defaults/web.xml is not realy needed, yet),
  but could be expanded
- lookup for web.xml - 1. in /DATA/SETTINGS then in /defaults

exclude terms with " for didYouMean suggestion
causes Solr error (and wordindex likely finds suggestion)

org.apache.solr.core.SolrCore org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse 'text_t:""d"': Lexical error at line 1, column 12.  Encountered: <EOF> after : ""
	at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:171)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:187)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.query(EmbeddedSolrConnector.java:179)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector$DocListSearcher.<init>(EmbeddedSolrConnector.java:345)
	at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getCountByQuery(EmbeddedSolrConnector.java:364)
	at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getCountByQuery(MirrorSolrConnector.java:326)
	at net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getCountByQuery(ConcurrentUpdateSolrConnector.java:440)
	at net.yacy.search.index.Segment.getWordCountGuess(Segment.java:464)
	at net.yacy.data.DidYouMean.getSuggestions(DidYouMean.java:181)
	at suggest.respond(suggest.java:73)

use configured admin-username for api calls
- the admin user name can be configured, in apiExec calls the default "admin" username is used. 

TODO: the bin/apicall.sh script should likely take that into account.

update to Jetty 8.1.14

refactoring (usage of constant names for attributes of authentication
check)

fix "add to blacklist" button style in IndexControlRWIs_p
- added default filename filter to select field (as only addition to *.black list is permanent)

- modified Blacklist_p header/legend to show all active blacklists 
  (to support understanding that all configured lists are active)
- removed obsolete code in Blacklist_p servlet

remove dependency from old serverCore.java
- remaining getPortNr not needed 
  (as current release allows only to set plain integer as port,
   see ConfigBasic)

added Windows Service installer
- used Apache procrun as it has a small footprint and comes with a GUI to edit the service settings
  see http://commons.apache.org/proper/commons-daemon/procrun.html

- added the service runner exe file under addon/windowsService
   as is (without renaming the exe files)
- included installYaCyWindowsService.bat and uninstallYaCyWindowsService.bat to main directory
  - which chooses the native exe according to the processor_architecture

upd: to open more external links in new browser-tab 

added SSL/HTTPS connector to support SSL/https connection on port 8443
!!! attention !!! to make sure YaCy can start, https will be disabled if port 8443 is used
   - added ping test for above to migration 

- as of now port for https is hardcoded to default 8443
- if not urgend required I'd leave it this way (it's standard) to use different ports for http and https 

- post https port on ConfigBasic.html (if active)

removed unused anomichttpd code after migration to jetty

migrated to solr 4.6.0

Merge origin/master into jetty

- added a SolrQueryResponse2SolrDocumentList method which is able to
work around the unfolding process in Solr's BinaryResponseWriter.
This was a huge performance bottleneck in the embedded solr connector
and the problem is actually on Solr side, but we have now a workaround.
- This made it possible to abstract a high-performance index access
method which is implemented as method getDocumentListByParams. That
method is also implemented in the SolrServerConnector and provides a
very efficient access to a solr index if the index is embedded.
- a popular use of the document list retrieval is a result count which
can now also make use of the new method, via getDocumentCountByParams.
- enhanced the Error cache which now does not store error documents
within the ram cache if the document is also written to solr. When
documents are retrieved from the cache, they are partly read from the
ram cache and if not existent there, from the Solr index.

Merge origin/master into jetty

enhanced webgraph processing

Merge origin/master into jetty

Conflicts:
	source/net/yacy/cora/federate/solr/SolrServlet.java

Merge origin/master into jetty

Merge origin/master into jetty

added option to Config_Network_p.html to enable remote search while
DHT-Receive is switched off.

set more logger to 'final static'

merge rc1/master

found and fixed a huge memory leak in solr caching (inside Solr). The
not-flushed Solr cache is now handled in this way:
- it is smaller by default
- an Solr-internal process is started to flush the cache periodically
(this does NOT clean the cache, just removes old objects)
- a Solr-external process (the standard YaCy cleanup-process) now has
direct access to the solr internal cache and flushes them completely.
The time frame for such a flush is defined by the cleanup-process
frequency, by default 10 minutes.

Merge origin/master into jetty

Speed enhancements and less CPU usage during Solr searches when using
the embedded Solr (the default). This was obtained by cirumventing solrj
search encapsulation and the implementation of direct index access
methods to Solr.
The effect will not only be seen during search, but this has also a
strong effect on suggestions (much more) and less CPU power usage during
index distribution (which needs many search requests)

Merge origin/master into jetty

hacks to prevent storage of data longer than necessary during search and
some speed enhancements. This should reduce the memory usage during
heavy-load search a bit.

- fixed a problem which ocurred when a document was not recognized with
the right content domain (i.e. identifying that it is an image, text
etc.) because it used the file extension and not an existing mime type
assignment.
- fixed the new setting that images shall be loaded for a better image
search.
- both fixes together makes it now possible to crawl
commons.wikimedia.org which makes use of 'funny' document names (i.e.
ending with .jpg while the document is html)

merge rc1/master

merge rc1/master

Merge upstream/master into crawlexpert-post

Merge master into crawlexpert-post

downgrade to Jetty 8 to assure support of JRE 1.6
- introduce a YaCyHttp interface to modulize/separate http server
- adjust the Jetty version specific implementation part (in package net.yacy.http)
     - putting the version specific code in classes starting with Jetty8xxxx
     - moved existing Jetty9xxx implementation into a test class (to keep the code)
- adjust build to the changed jars
- make use of the introduced YaCyHttpServer interface in related htroot servlets

- adjust other test cases/classes

migration to Solr 4.5.0

merge with rc1/master

Merge remote-tracking branch 'origin/master'

Conflicts:
	.classpath

self-healing of mistakenly deactivated crawl profiles. This fixes a bug
which can happen in rare cases when a crawl start and a cleanup process
happen at the same time.

bump to httpClient-4.3

added the new field harvestkey_s to the collection index and the
webgraph index which is temporary filled with the crawl profile key.
This is used to select a set of documents for post-processing as soon as
a crawl is finished. Now the postprocessing for a specific crawl is
started when that specific crawl is finished and not at the end of all
post-processing steps.

merge rc1 master

Merge remote-tracking branch 'origin/master' into jetty

- removed ZURL data structure; removed also the ZURL data file
- replaced load failure logging by information which is stored in Solr
- fixed a bug with crawling of feeds: added must-match pattern
application to feed urls to filter out such urls which shall not be in a
wanted domain
- delegatedURLs, which also used ZURLs are now temporary objects in
memory

- replaced the properties object in AnchorURL with distinct variables
for anchor attributes.
- this caused that large portions of the parser code had to be adopted
as well
- added a counter target_order_i for anchor links in webgraph
computation

- the webgraph shall store all links which appear on a web page and not
all unique links! This made it necessary, that a large portion of the
parser and link processing classes must be adopted to carry a different
type of link collection which carry a property attribute which are
attached to web anchors.
- introduction of a new URL class, AnchorURL
- the other url classes, DigestURI and MultiProtocolURI had been renamed
and refactored to fit into a new document package schema, document.id
- cleanup of net.yacy.cora.document package and refactoring

update to Jetty 9 jars
- include javax.servlet 3.0

Merge remote-tracking branch 'origin/master' into jetty

added parsing of 'date', 'dc:date', 'dc.date' and 'last-modified' in
html meta fields to get a correct (or: better) date timestamp. The
http:last-modified mostly does not work because it is set to the current
date from most CMS.

merged current rc1/master into jetty branch to allow further development with latest version
ServerSideIncludes and servlet return values need further work (for working jetty integration)
- TODO: added nasty quickfix to allow SSI -  needs further work
- TODO: YaCy servlet return values/parameters are not handled

added two new fields, exact_signature_copycount_i and
fuzzy_signature_copycount_i, which count the number of copies of
non-unique documents and assigns this to each document. Thus, each
document there is a number assigned which shows how many copies of this
document exists.
These fields are disabled by default.

redesign of the image search process (with much better results,
unfortunately the index schema has changed and p2p image search will not
be muchmuch better until many people update)

added a feature to set a collection for a crawl result based on a
regular expression on th url: the collection attribut for a crawl start
may be now either a token or a list of tokens, seperated by ',' where a
token is either a string or a pair <string,pattern> where the string is
separated to the pattern with a ':' and the string is assigned to the
document as collection only if the pattern matches with the url.

Redesign of crawler identification and robots steering. A non-p2p user
in intranets and the internet can now choose to appear as Googlebot.
This is an essential necessity to be able to compete in the field of
commercial search appliances, since most web pages are these days
optimized only for Google and no other search platform any more. All
commercial search engine providers have a built-in fake-Google User
Agent to be able to get the same search index as Google can do. Without
the resistance against obeying to robots.txt in this case, no
competition is possible any more. YaCy will always obey the robots.txt
when it is used for crawling the web in a peer-to-peer network, but to
establish a Search Appliance (like a Google Search Appliance, GSA) it is
necessary to be able to behave exactly like a Google crawler.
With this change, you will be able to switch the user agent when portal
or intranet mode is selected on per-crawl-start basis. Every crawl start
can have a different user agent.

- refactoring
- generalized writing of url attributes to solr documents
- added more url attributes to error documents

fix for default boolean post values: the default value MUST NOT be TRUE,
because it's normal that a boolean value is missing in the post argument
if a checkbox is not selected.
Added also some style enhancements to IndexFederated, removed the Solr
attachment manual and replaced it with a link to the wiki which explains
this in more detail.

migration to solr 4.4.0

replaced the single-text description solr field with a multi-value
description_txt text field

catch solr close exceptions

enhanced memory protection and OOM exception handling in Solr connector

fixed setting of shortMemoryStatus in MemoryControl

fixed 'resource leak' warning

fix vocabulary navigator checkbox selection (from last commit)

fix auth for forced ping

fixed missing corona in network picture

fix for bad-formatted lonlat

fix for http://bugs.yacy.net/view.php?id=354

fix for OOM during remote search and too high load protection

removed debug code

NP-fix, which was found on a Pi under 'havy' load

fix role name in GSA servlet

fix: NPE on shutdown via script 

fix: display proper error msg on admin user change

fix double-escaped urls from proxy-usage

fixed shutdown authenticaton problem

fix  external link (open in new tab)

added security patch

wrong logger fix

alphabetic order fix

fixed the SolrServlet

next try to fix m4

fix for deb m4

fix for bad ErrorCache.exists test (bug from latest commit)

IPv6 fixes

fixed bug with image search in yacyinteractive

fixed urlmask which was not able to combine several constraints

fixed a NPE in surrogat processing

fixed a division by zero

fix NPE on mime detection of unknown file extension

include/exclude size NPE fix (recently added)

fixed seedlist (hash was missing)

fixed json

purge a lucene cache - possible memory leak fix

fix for another XSS bug

fixed some XSS bugs reported by Marius from http://ctf365.com/

fix for wrong counter

fixed and enhanced postprocessing

fixed termination bug in Solr Connector

enhanced and fixed postprocessing

fixed several problems in solr connectors

fixed keepalive min value

fixed logging for remote solr configuration

removed debug code for existsByIds

added debug code

URL-export:add query and fix for cast-class-exception

fix in html parser

fix typo in GitRevTask (branch)

fixed sitemap crawl start

fixed webgraph postprocessing (did nothing and repeated to do this...)

bump to httpclient-4.3.1 - a bugfix release

fix last commit (not needed test of GZipInputStream)

NPE fix for my last commit

fix for didyoumean, added also more asian alphabets

enhanced ranking; patches to replace old ranking

remove debugging code intercepting post requests

fix typo in parsing "cachePolicy" parameter

Fixed template vars for "deleteold". Fixed parsing "deleteold" parameter. Stop "setState" overwriting "deletold" state on load.

fixed wrong parameter check

Use list template for all other option lists. Fixed some template expressions.

fixed search navigation

fix for rss reader, see http://bugs.yacy.net/view.php?id=294

fix : forgot adding security handler

fix NPE on modified since check ( Response.requestHeader allowed to be null)

added classpath for debugging

fix for canonical-from-http-header feature

fix for crawl domain counter limitation (limit was reached too early)

fix not necessary use of DigestURL

fix for npe during location search

fix for 100%CPU problem in error cache cleaning process

fix for parsing of image links inside of anchor links (image-links)

added read parameter from multi-part form fields  (to nasty quick-fix)

fixed wrong facet count

fix for content domain classification in URIMetadataNode

fix for catchall query goal for image search

fix for dc:description field

one more fix for image search

fix for solr/gsa result logging

npe fix

NPE during shutdown fix

fix for http://bugs.yacy.net/view.php?id=286

fix for bad snippets in gsa api

html fix

info msg if no embedded Solr http://bugs.yacy.net/view.php?id=279

css fix for IE10 showing  border on all img within <a /> tag since introduction of external link icon (commit https://gitorious.org/yacy/rc1/commit/112836dcc9bfd3966a81f2a6160569e69a256a40)

bugfix release 1.62

added line to enhance debugging

fixed feed api servlet and and enhanced RSSReader class

fix for CGI test

fix for Blacklist (-Administration)

fix for npe

Release 1.68

extra sleep for remote searches enhances search results because there is
more time for more remote peers to contribute on the first result page

more efficient wordcount

added memory protection for postprocessing

removed warnings and superfluous logging

removed interaction fragments

fixed "Size in KBytes" calculation in PerformanceQueues_p.html,
see http://bugs.yacy.net/view.php?id=362

removed unused classes and servlets

removed libraries which had been used by jena and were added together
with jena in commit 5d20cd324af387c72fedec88a8059442c090c769, see:
https://gitorious.org/yacy/rc1/commit/5d20cd324af387c72fedec88a8059442c090c769

enrichment of PerformanceMemory display of SolrInfoMBean table

added statistic information for solr resources in PerformanceMemory

fix: select all checkbox Tables_p 
(needs form name attribute)

re-inserted default value re-set button to performance queues and
patched missing values for recent new queues

updated and completed translation of Steering.html and
ConfigUpdate_p.html into French

enhanced strategy to clear solr caches
- redesigned the instance mirror class (which was a mess)
- added final method to close a searcher (which otherwise keeps a cache)
- changed cache clear method which iterates over resources and calls
clear to all caches in the searcher resources

updated and completed translation of Network.html into French

translated templates into French

fix not needed getFileExtension().toLower (double)
add missing .getFileExtension

completed fr.lng for Status.html

completed fr.lng for BasicConfig.html (+modif in de.lng)

enhanced didyoumean

enhanced didyoumean

reduced color depth of star icons (for smaller file sizes)

small change to start/stop/pause icon style

added new suggest method which replaces more-than-one suggestions:
instead of computing suggest permutations of the given words, the
completion of a phrase using the given words is searched in the fulltext
index.

remove unused static var from HTTPDProxyHandler

allow combining  selection of different search nav's (facets)
- selecting more than one nav combines the 2 selections (with AND)
- unselecting one nav clears all selected

(e.g. select filetype:pdf and /language/fr shows ~ french pdf's only)

fix page nav, to keeping modifier 
(was new issue)

added a boost on appearance of combined words (in the same order the
user submitted that) when searching for more than one word

URLproxy access check too tight 
respect config ip pattern (was own ip)

- faster initialization of index files
- removal of not used space if index files shrink (rare, but possible)

less logging

updated bootstrap seedlist list

enhancements for staticIP and ipv6 handling

add config value for ssl/https port (default=8443)
adjust server routines to use config

disable wrong forward to https on port change

improve isOlder check to not overwrite node index with metadata on equal load date

remove double jquery-1.3.1.js and adjust header links to jquery-1.3.2

add language navigation filter clause to rwi results

added hint to web.xml and for completeness the full set of hardcoded mappings

Added a write-enabled checkbox for remote solr servers.
It is now possible to assign every peer other YaCy peers as remote solr
server which are only used for read operations during search. This also
affects crawling: it will exclude urls from crawls which exist on remote
solr/remote YaCy peers.

changed solr paths according to new default paths

first try to add a generic solr servlet as luke request servlet

changed the web.xml as well to migrate the solr servlet

refactoring: renamed the SolrServlet to SolrSelectServlet for better
naming of more Solr Servlets

added two virtual instances of solr for the both cores: collection1 and
webgraph. These cores are now accessible at
/solr/collection1/select instead /solr/select?core=collection1
and
/solr/webgraph/select instead /solr/select?core=webgraph
in addition to the old behavior to support compatibility to the old
peers. These new paths are fully solr standard-conform and will allow
the cross-linking between YaCy peers using their public solr API.

added button in host browser to re-load 404/failed documents

fix search navigator not showing activated nav
 introduced with https://gitorious.org/yacy/rc1/commit/97e84439fb749a624c00ec01a6caa076c9c1c32f

optimize parse ServerSideInclude

use less ram on dht-URL transfer by not using a URIMetadataNode[]

do not set a load prerequisite that prevents the start of one-time-jobs

workaround to let dht-out run on smal Systems like a Pi

put  list quick navigator buttons belowon BlackList_p editor
replacing the dropdown -> go navigation

making latency_factor and maximum number of same hosts in loader queue
settings available in Crawler_p.html servlet for steering.

code simplifications / removed warnings

new crawling factors can now be changed during runtime

- removed hardcoded load-test which is now handled in BusyQueues
steering, see /PerformanceQueues_p.html
- changed default values for crawler queue load limit (high, because
these jobs are started upon user request)

lower client-connection for single-cpu-systems

reverted crawling strategy from latest commit.

enhanced crawler speed

added synchronization to exists() call bacause the concurrent call to
that method showed in thread dump close to deadlock situations. Its also
better to synchronize IO operations because they become faster then.

if a http client thread is still running but we do not wait for it any
more, call an interrupt

cleanup crawl loader jobs

fix IOexception logging issue in DefaultServlet
reason not sure but .logException triggers another exception

upd heuristic default list (heuristicopensearch.conf)
- Faroo Web taken out (requires api key) http://www.faroo.com/hp/api/api.html#description
- update Faroo News to new url
- Twitter taken out (change to Api 1.1 not supporting rss) https://dev.twitter.com/discussions/24239

experimental removal of synchronization. This should work for all cases
where the size() and isEmpty() method is used only for statistics, which
happens at many locations in YaCy. If these methods are used for
structual reasons (like accessing the last element in an array) then it
may fail or cause other problems. As far as visible, this is not the
case.

removed superfluous synchronization

added load limit for DHT transfer: RWI acceptance only if local load is
not too high

fix bug: Crawl Start (Expert) crawls "?-URLs" even if told not to do so
http://bugs.yacy.net/view.php?id=329

Skip remote Solr search if last try showed error
As the solr servlet may not be available (e.g. no public search page, old version, individual access setting) a /solr/select error is 
remembered in the seed.dna of the remote peer.
This is not permanent, as flag is not stored and the seed is reloaded on several occasions, it is just a memory of the recent past status.
Might also be set to "not available" on time-out of last try.

concurrency-solid version of transmission limitation

removed -d64 jvm option because that causes problems on non-64 bit
linux, see http://bugs.yacy.net/view.php?id=349 and
http://bugs.yacy.net/view.php?id=339

reduce load during dht transfer

no commit if no action is requested

- better 'extra'-peer selection
- logging of health status for 'extra'-peer selection
- concurrency for remote peer IO and interrupting the threads if
time-out occurrs

added synchronization to deeply-embedded solr connector
EmbeddedSolrConnector because deadlock situations show that methods in
lucene class seem to block.

when scaling the number of remote peers, also consider the machine load
and the number of cores

less blocking in SeedDB which caused deadlocks in peer ping

better logging in logger

replaced old caching in SolrConnector with a new one which is better for
concurrency and should prevent from 100% CPU usage after a long run of a
peer with a large number of documents.

backmigration from solrj 4.6.0 to 4.5.1. This is necessary because
solrj.4.6.0 has a bug which prevents the attachment of a remote solr (as
tested with a SolrCloud). See bug report
https://issues.apache.org/jira/browse/SOLR-5532
This bug shall be fixed in Solr 4.6.1.
Fortunately, solrj-4.5.1 works together with solr-4.6.0 thus the current
index does not need to be changed.

simulate Authorization cookie for yacy servlet header

fix NPE in TemplateEngine

StackTrace For input string: ""
java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:504)
	at java.lang.Integer.parseInt(Integer.java:527)
	at net.yacy.server.http.TemplateEngine.writeTemplate(TemplateEngine.java:241)
	at net.yacy.server.http.TemplateEngine.writeTemplate(TemplateEngine.java:199)
	at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:896)

implementing YaCy legacy role names
- taking out customized SecurityHandler code as the original/default seems to just work fine
- with this individual sec. constraints can be applied via web.xml (using legacy role names)

Merge origin/master

make SecurityHandler  webappcontext ready

added checkindex, solr index check

disabled unused fields in default Solr collection schema

spend CrawlQueues a fremem routine
- clears errorStack
- will not get hit often (but better little than nothing on low mem)

fix use of url proxy access pattern
pattern of transparent was used.

Merge origin/master

added a script which can check the solr index for inconsistencies while
the peer is down. This shall be used in emergency cases where a check or
fix for a broken solr index is needed.

using the adminAccountUserName from yacy.conf within apicall.sh

introduced a new storage path ARCHIVE inside of DATA which will be used
as path for solr index dumps (instead of the SEGMENTS path). This will
make a maintenance of index backups easier. It will also provide a tool
to migrate from an freeworld index to a webportal index.

check blacklist proxyClient config

make use of our DNS-cache again - this realy speeds up the lookup

better solution for prev. commit with MultiMapSolrParams.getFieldInt not returning default parameter

improve NPE chance on missing parameters
java.lang.NullPointerException
	at net.yacy.http.servlets.SolrServlet.service(SolrServlet.java:145)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)

reimplement proxy access by configured whitlist pattern
was currently limited to own ip.

setting of IPv4Stack moved earlier
it seems even better to call system.setproperty before isrunning check
(if nothing helps we have to set it in startup script)

open service manager upon install failure in installYaCyWindowsService.bat
likely service is already installed
Service Manager allows then to modify settings

reimplemented server access restriction
via Jetty IPAccessHandler to allow only configured IP's to access.
Handler is only loaded if a restriction is configured.

Since IPAcessHandler (Jetty 8) does not support IPv6 system property java.net.preferIPv4Stack=true
Testing showed system.setProperty seems to be sensitive to point of calling (earliest possible time seems to be best = early in yacy.main).
Moved the "isrunning..." just open browser check also to the new routine to preread the yacy.config only once.

set a realm message to log-in input window which explains that a
password for the account 'admin' can be (re-)set with the script
bin/passwd.sh

cleanup

removed non-working realm setting in http client (auth for localhost was
added in previous commit)

added http authentification to YaCy http client for all localhost
acesses to enable self-steering of the peer using the API table. This is
necessary in case that an password for the administration pages is set.

modified log-in detail to enable admin-login from localhost with stored
hash even if localhost access is disabled. This is urgently needed for
the apicall.sh script since that is used for high-availability set-up
(checkalive and indexdump for index mirroring)

made the access tracker class static because it shall be used by the
jetty auth module

fixed mess with test on localhost (which means local hosts for some
cases)

removed the & explicit encoding of ampersand character since this is
double-translated within the template replacement process.

modified DefaultServlet parameter on invoke templates
call response with post=0 (if post empty) simulating previous behavior.

(template servlets typically test for post==null,
found one more Crawler.p.java were empty post caused problem,
= defaults not correctly set)

fix opensearch resultcount in yacysearch.rss
see merge request https://gitorious.org/yacy/rc1/merge_requests/24
use result count in searchtrailer.xml which is on p2p search more accurate (timing)

adding proxy status to (private) status box
(show also transparent and url proxy status)

show search result via url proxy only if status=on

handle http connect for proxy
refactor header cleanup (reuse existing code)

harmonize access restriction for urlproxy servlet
with proxy handler, what is currently
- use switched on in config
- access from a local IP / hostname

fix shutdown exception for crashprotection handler on interrupted connections.

extend ssl enabled msg on status page 
- post the portnr

extended request for local peer check to by hostname resolved ip
the current islocal() check did not detect a domain.com address as request for the local peer.

add error response on deniedl  proxy access
send http 403 response

reimplement proxy access log (to Jetty ProxyHandler)
- using existing HTTPDProxyHandler logger
- allow local loopback ip to access proxy

Security Hot-Fix: for transparent proxy.

add to blacklist button in CrawlResults
http://bugs.yacy.net/view.php?id=220
introduced Blacklist.add with sourcefile only parameter

moved startup execAPIActions call after Jetty startup
execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, 
but it's more robust to place the call after http is assigned to switchboard/serverSwitch.

change SolrServlet from Filter to Servlet
(as no multicore required)
this allows to simplify context/servlet initialization in Jetty init.

fix NPE GSAresponseWriter on query=null
java.lang.NullPointerException
	at net.yacy.cora.federate.solr.responsewriter.GSAResponseWriter.highlight(GSAResponseWriter.java:328)
	at net.yacy.cora.federate.solr.responsewriter.GSAResponseWriter.write(GSAResponseWriter.java:263)
	at net.yacy.http.servlets.SolrServlet.service(SolrServlet.java:235)

new data type for access tracker (unfinished)

include install/uninstallYaCyWindowsService.bat 
as wrappers to copy

fix QueryGoal Image query (missing space)
see query log example .. url_file_ext_s:(jpg OR png OR gif) ORcontent_type:(image/*)) ..

unescaping the path

"inaccessible" better describes the state of a server which cannot be
reached (while 30c3: too many users)

fix NPE  in IndexControlRWIs_p.html
- metatags my be null
Caused by: java.lang.NullPointerException
	at net.yacy.search.query.QueryParams.getFacets(QueryParams.java:445)
	at net.yacy.search.query.QueryParams.getBasicParams(QueryParams.java:400)
	at net.yacy.search.query.QueryParams.solrTextQuery(QueryParams.java:345)
	at net.yacy.search.query.QueryParams.solrQuery(QueryParams.java:334)
	at net.yacy.search.query.SearchEvent.<init>(SearchEvent.java:290)
	at net.yacy.search.query.SearchEventCache.getEvent(SearchEventCache.java:176)
	at IndexControlRWIs_p.genSearchresult(IndexControlRWIs_p.java:641)
	at IndexControlRWIs_p.respond(IndexControlRWIs_p.java:141)

Merge origin/master

upd NB gui classpath

switched back on index deletion (was accidently off because new jetty
framework delivers never null to post arguments .. there may be more of
that kind of problems)

remove in distro not existing path setting from stopYaCy.bat

write solr searches to search log

fix for strange NPE
java.lang.NullPointerException
        at
net.yacy.search.Switchboard.updateMySeed(Switchboard.java:3667)
        at net.yacy.peers.Network.peerPing(Network.java:195)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:107)
        at
net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:165)

startRecord fix; this is not according to SRU definition because this
states that the first record has number 0; but +1 is not consistent with
other places where the number is used.

ftp client shall be able to open non-anonymous ftp servers if login
details are given

display configured adminUserName in ConfigAccounts_p
- fix read default username in  in loginservice

Merge origin/master

tweak Jetty credentials to work with YaCy UserDB
- user entry in UserDB with admin right can login to access protected pages
- dto. admin user, choosen username is stored in conf (adminAccountUserName=)

Merge origin/master

nasty quick fix for admin login with other username as admin
- userDB is not sync'ed with Jetty credentials as of now only the std. admin account can login

switched initial browser open with ssl active back to std. http port

added option to attach remote solr servers in read-only mode

add patch which shall prevent that naive search mistakes like usage of
regular expressions cause no results. Usage of '*' followed by a dot or
any expression will now cause that this expression is used as a filetype
search.

- use default files from yacy.init property "defaultFiles" if no
jetty-configuration is given for default files.
- fix a problem with default paths if no path is given (i.e.
http://localhost:8090 instead of http://localhost:8090/). Without this
patch the path was resolved automatically to http://localhost:8090//

suppress deprecation warning (for now); TODO: find alternatives

removed classpathentry to 'remove obsolete htroot/solr htroot/gsa
YaCy-servlets'

missing entrys for: 'updated poi-3.9 / poi-scratchpad-3.9'

upd installWindowsService.bat
- remove javaopts according to startYaCy.bat

remove obsolete htroot/solr htroot/gsa YaCy-servlets
- now handled by standard servlets

update Maven pom

updated poi-3.9 / poi-scratchpad-3.9

run SSL/HTTPS port (8443) ping test in migration only if SSL/HTTPS is on
- see last commit

remove obsolete HTTPServer input field

aaaaaaaaaaaaaa

remove large memory pages option and added MMapDirectoryFactory for
debian

added html retrieval from alternative DATA/HTDOCS path

removed more unused code

refactoring: moving all servlets to the same package (the solr servlet
is currently actually a filter which should be changed somehow)

solr servlet preparation for join with jetty branch

updated guava to 15.0

upgraded json-simple to 1.1.1

upgraded jsch to 0.1.50

removed mysql connector which was used only for testing in the past

migrated to pdfbox 1.8.3

added patch to allow binary response writer

update to Mac start parameters

removed malicious java start arguments in debian start script which had
been removed from the other start scripts recently. The reason to do so
was a comparisment of a debian-installed YaCy with 20 million document
which crashed after 10 hours with the debian start script, but did not
crash with the startYACY.sh start script. Both scripts now use the same
java start arguments.
Added also the Solr MMapDirectoryFactory switch which was missing so far
in the debian start script.

removed non-root restriction in startYACY.sh because that is also not
applied for the debian version. There is no reason to apply this
restriction.
Added also the -d64 option if the system is 64 bit.

added parsing of metadata to surrogate reading:
a dublin core record inside of surrogate input files may now contain
tokens within the namespace 'md' (short for: metadata). The token names
must be valid withing the namespace of the solr field names. All
md-tokens inside of surrogate files then overwrite values within solr
documents before they are written to the solr index. This makes it
possible to assign collection names to each surrogate entry and also
ranking information can be added. Please see the example file.

remove unused init parameter from DefaultServlet
- remove "RelativeResourceBase" parameter

using stringbuilder in query construction

update Netbeans GUI build classpath

use pre-compiled patterns in ymark

made pattern computation static

added date parser caches to prevent re-calculation of costly date
parsing

counting search requests in solr interface

avoid usage of existsByQuery. If a document can be loaded by the ID
before testing other fields from the existsByQuery request, then a
document cache fills and queries after that one can be avoided.

join DefaultServlet and Jetty8 implementation 
- removing Jetty 8 specific dependencies

update Maven pom to latest version number
- include newer dependency versions of several lib/jar for eval.

move conditionalHeader to DefaultServlet
- by removing Jetty specific implementation detail

added more properties to seedlist servlet

next development version

changed start parameters which caused deadlocks in mac and windows
versions

intermediate release 1.66

better calibration of caches and queue maximum sizes

perform greedy learning heuristic only if load < 1.0

changed default timeout for port scanner

removed unnecessary check which causes a NPE when searching with empty
search string

- reduce computation in case that specific postprocessing fields are not
selected
- de-select citation rank computation

removed optimization bevore postprocessing because that may cause a
time-out which will cause that postprocessing fails.

removed filter computation in Protocol class for remote searches because
that is already done in the QueryParams class

- performance graph does not shop peer ping in memory monitor any more
- after a forced GC, the PerformanceMemory view switches to automatic
update by default

clean-up Jetty handler classes

clean-up test cases 
to work with current source

add read queryparameter in gsaservlet

remove reference to solrServlet from YaCy servlet select
- reference is not used
- solrServlet is used in Jetty branch and adjustments there conflict with unused solrServlet here.

tried some hardening in order not letting any Solr-Searchers open

setting of some default values to make an client development start easy
using the description at
http://www.yacy-websuche.de/wiki/index.php/Dev:APIhello

tried to fix timeout and connection-lost problems when using an outside
solr.

try to recover from an OOM during citation index reading and fail-over
to second solr core in case of unrecoverable OOM.

fixed a problem with Date fields parsing Solr results if a remote Solr
is attached.

memory-leak-fix: the DocListSearcher fires an query in its constructor
and it is highly recommend to close every SolrRequest.
Every Request, which is not closed leaves a Searcher with its Chaches an
can not be garbage-collectet.

implement core selection in SolrServlet
- making initcore() obsolete

disallow remote robots to crawl the HostBrowser servlet

only admins should be able to force a commit

fixes for searches containing stopwords. The fix was done using a
reconstruction of the search word set access method to protect that
words are deleted from the sets from the outside of the QueryGoal class.

hack to remove compiler warnings about deprecated classes. It would be
better to remove the deprecated usage but to do this the Solr core must
adopt the latest apache http core changes as well .. this is not our
fault.

using the BinaryResponseWriter which is supported within the YaCy solr
servlet since YaCy 1.63. This is much more performant for the client
than using the XMLResponseWriter because parsing of XML data is very CPU
intensive. Older YaCy peers are still requested using the
XMLResponseWriter but the majority of YaCy peers already respond with
the binary writer. This makes remote searches much faster and less CPU
intensive.

less word hash computations (removing some overhead because of MD5
calcs) using the clear word in a normalized form.

add check to prevent index entries containing url_file_ext_s with ";jsession=xyz"
note: check could be implemented in MultiProtocolURL (but at this time didn't oversee possible implication)

remove unused options and attributes from DefaultServlet
cleanup obsolete class files

fix contentType definition for Solr html responswriter
from xml to html
(hint: value is currently not used, but is in SolrServlet)

refactoring of seed attributes (introduced more constants)

added test to Base64Order (runs successfully!)

fir for XSS

 added new servlet which generates the same file as the principal peers
upload to a bootstrap position
 you can call it either with
 http://localhost:8090/yacy/seedlist.html
 or to generate json (or jsonp) with
 http://localhost:8090/yacy/seedlist.json
 http://localhost:8090/yacy/seedlist.json?callback=seedlist

for searchresults with empty title use filename as title
- to not store a title in index which isn't extracted from source 
  the title is empty check only added to ResultEntry class

- disable Jetty servlet defaultUseCache (prevent double caching)
- include short memory status check for class cache in DefaultServlet
- remove obsolete Resource interface for Jetty8YaCyDefaultServlet

use writeOptionHeaders with ServletResponse parameter only

extended also timeout fr webgraph postprocessing

extended timeout during postprocessing of 30 minutes.

more monitoring for postprocessing and enhanced layout in Crawler
monitor page

more monitoring for postprocessing

prevent requesting of 0-facets (which actually exist)

enhanced mass storage speed to remote solr servers

added an optimization option to HandleSet mass data storage structure

- DefaultServlet: remove static gzip option
    YaCy doesn't use pre-gzip'ed static html pages 
- ProxyServlet: remove not neede procedure
- Server init: skip one overlaping servlet context

logger should be static

fix for search requests in GSA interface which contain 'funny'
characters (like ':' etc.)

replaced old existsByIds for embedded Solr with obviously much faster
new selection method (including stil existing debug code to test that
this is in fact better)

make test directory available in Maven pom
- exclude reference to old slf4j-log4j12

cleanup test directory from Jetty 9 implementation samples
- current Jetty implementation advances so that it seems not beneficial to keep the code
as it makes the test unuseable and use of Jetty 9 is due to Java 1.7 dependency not in sight.

implement Jetty8HttpServerImpl.generateSocketAddress
(code 1:1 copied from serverCore)

added (partly commented) test code for url rewrite methods .. to be
completed

better default value for crawler user agent

add highlight queryparams to solrservlet if missing
- modify query params in Solr parameter map (instead of querystring)

reverted autowarming setting in solrconfig

Merge origin/master into jetty

Conflicts:
	source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java

calculating and showing the number of files and the average size of a
file in the HTCACHE in ConfigHTCache_p.html

fixed another memory leak in the PDF parser:
the class org.apache.pdfbox.pdmodel.font.PDFont occupies 8MB of space
which cannot be cleaned if PDFont.clearResources is called.
The attempt to clean the class cache therefore causes that the class is
loaded and this cache is initialized with some rubbish. I tried to
prevent to instantiate this class by usage of a hacked findLoadedClass
call to the SystemClassLoader (which is protected ...).
Now, without using the PDF parser at all, 8MB of RAM space is not
occupied, however, when the first PDF arrives this space will be taked
and never given back to GC.
WAKE UP YOU LAZY PDFBOX HACKER AND FIX THIS SHIT!

reduced footprint of BookmarksDB which can take quite a lot of memory if
the number of bookmarks is high (i.e. > 2000 URLs)

add content_type:image/* to image search
- see numerous idx entries with content_type image without url_file_ext_s (for various reason) which should be included in result
- try it yourself with following sample query
   /solr/select?q=content_type:image/* AND -url_file_ext_s:[* TO *]&defType=edismax&fl=sku,url_file_ext_s,content_type

adresses also possible url without or deviating extension.

move writeHeaders from Jetty8 servlet to YaCyDefaultServlet
- after removing Jetty server dependency (of Response using HttpServletResponse only)

replaced referrer link masking to 'pure' links to the referring page
(that was more useful during testing)

added missing unicode transformation in href link contents during
parsing

added clickdepth and CR computation debug code to verify that the
process is complete

we do not need the load-image flag any more since this is now controlled
by parser switches

add AccessTracker logging to SolrServlet

implement HtmlResponseWriter to solrServlet (and rss / opensearch responswriter) as in yacy select servlet.
- set contenttype of HTLM/GrepHTML-Reponsewriter to "text/html"
- set a contenttype to GSAsearchServlet

load image only if their parser flag is activated

less overhead calling exist() with only one hash

fix html header in Solr HTMLResponseWriter
- move 1st body content after </head> tag
- add closing <span> tag

implement a Jetty reconnect to work with Configbasic_p.html port change
- instead of shutting down the server it should be sufficient to manipulate the Jetty http connector

avoid cloning

upd Maven pom (take back introduced java-templates)
to handle filtering of yacyBuildProperties.java.
To keep it compatible with ant filter directly from original sourcd/.... location.

fix servlet class exist check to use default path only (in Jetty8YaCyDefaultServlet)
- del redundant doget code in yacydefaultservlet
   - small declaration code opts
- del obsolete libt/proxyservlet.java

update Maven build script
- use current YaCy version number
- make use of libbuild\GitRevMavenTask (maven-plugin-gitrevisionnumber)
- make yacyBuildProperties.java available for source filtering by Maven-plugin (copy to libbuild\java-templates)
- update assembly definition to include lib\yacycore.jar without version number (needed this way by startupscript)

add Maven plugin to  return a YaCy style Git repository build release number and timestamp
- it injects properties which can be used in pom via ${DSTAMP} ${releaseNr} if added as plugin via
<plugin>
<groupId>net.yacy</groupId>
<artifactId>maven-plugin-gitrevisionnumber</artifactId>
<version>1.0</version>
<executions><execution>
<phase>initialize</phase>
<goals><goal>create</goal></goals>
</execution></executions>
</plugin>

in host browser also show the properties of failed documents including
referrer urls (this is a VERY USEFUL SEO and Web Admin feature!!)

version number change for next development cycle

use html encoding for urls in metadata

- add GSA search /gsa/search servlet for Jetty to Server init
- include SecurityHandler check for /gsa/ /solr/ 
- change one more YaCyDefaultServlet dependency from Jetty to std. javax.Servlet

update Maven pom and jar dependencies 

reduce logging (by assigning logger to existing logger)
- small additional cleanups

reactivate Domain handler for .yacy / .yacyh handling

resolve merge conflict
- add missing import statement 

Merge origin/master into jetty
- merge conflict resolve

add YaCyProxyServlet to handle /proxy.html?url=proxyurl
- based on Jetty ProxyServlet
- at this time use existing HTTPD ProxyHandler  for url rewrite
- add jetty-client jar (dependency in Jetty ProxyServlet)

reuse ProxyHandler.convertHeaderFromJetty in YaCyDefaultServlet

catch one more malformed url in proxy url rewrite

catch proxy url rewrite exception
 malformed url (" http:\/\/" ) may cause error response
 testcase http://localhost:8090/proxy.html?url=http://dictionary.reference.com/browse/test

next intermediate release 1.64

- fixed opensearchdescription, this delivered an url with missing
'global' option
- added display=2 to compare_yacy to remove the superfluous border

integrated startpage to compare_yacy

added boolean load property to CacheResource_p servlet which causes that
the servlet loads the page from the web.

- removed a lot of garbage and bloated code from GuiHandler.
- transformed log lines to String before they are stored because the
storage space is about 1:250 (45kb for one line before transformation,
180 bytes afterwards)
- this saves up to 10MB RAM so we can increase the number of lines to
1000 again.

Automatically flush the log cache if a short memory status is reached.
For the default of 200 lines this can flush about 10MB.

Enhanced crawl start for very, very large crawl lists (i.e. > 5000)
which had a problem because of badly used concurrency.
This fix also caused a redesign of the whole host deletion process.
This should fix bug http://bugs.yacy.net/view.php?id=250

reduced logging line memory, 10000 lines had filled up 450MB! grrr.
(thank you, a bomb from the past)

enhanced computation of number of solr documents.

added concurrency for mass crawl check

When doing bootstraping, always accept one seedlist-File without
checking the date of the file. This should help to start the peer in
case that the user has a completely wrong date setting.

replaced old tag cloud and adopted design a bit

added matching in url field in Table_API_p search

added hint to development/commit rss feed

Show a RSS icon in the right top corner of search results. This replaces
the 'API' icon which was the link for the opensearch result which is an
extension of RSS. Since it is more appropriate to visualize a RSS link
with an RSS icon, this API icon was changed here.

- prevent that crawl filter have empty (never-match) content
- rewrite the description of the options "Restrict to start domain(s)"
and "Restrict to sub-path(s)" to an explanation, that the restriction
applies to all links in the link list of the option "From Link-List of
URL" if this option is selected
- allow "Restrict to sub-path(s)" if the "From Link-List of URL" is
selected. This is supported in the crawl start.

fix for mustmatch regex computation: result had correct semantic, but
may have contained multiple same expressions within the disjunction of
domain-restrictions. This fix removes the redundant restrictions and
makes the regex shorter.

add graceful Jetty shutdown option
- as Jetty stop is not synced, yet
- include jetty jars and servlet-3.0 api jar  in Eclipse .classpath

make it easier to filter properties (case insensitive)

allow to choose used http server, YaCy-Anomic or Jetty
- defaults to Jetty (in this branch)
- add server version info & config option -> Admin Console -> Advanced Settings -> Http Networking

add YaCy HttpCommand "authenticate" check to DefaultServlet 

enhanced postprocessing: fixed bugs, enable proper postprocessing also
without the harvestingkey, remove crawl profiles after postprocessing,
speed-up for clickdepth computation.

make use of declared static string ACTION_LOCATION

add YaCy HttpCommand  "location"  check to DefaultServlet

more methods for the table class

Replaced the inframe loading from yacy.net for donations with the
loading of this iframe from the local host. To make this more flexible,
this iframe is loaded once after startup from yacy.net.

remove wrong content mod in SSI parser for virtual path /currentyacypeer/
(is handled on start of request handling)

add support for gzip encoded multipart forms (needed for transferRWI.html)
- quick and dirty reuse of existing HTTPDemon implementation

- add size check to multipart form data handling of YaCyDefaultServlet (same as in HTTPDemon.parseMultipart)
- reduce Jetty logging 
- give build.run a bit more memory (set to YaCy.default 600m from 512m)

- add localpeername.yacy to list of local address detection for AbstractRemoteHandler 
- use proxy via header info as in legacy proxy handler

fix YaCy version string in proxy headers
(config parameter vString not longer used)

disable domainhandler due to error
- domainhandler causes closed response output stream in following handlers 
  on addresses resolved to local peer (like in hello protocoll preventing peer to switch to senior peer)

update Info.plist
small DefaultServlet refactoring

searchpage javascript (latestinfo) causes reset of search statistic after moving to next page
- disabled call via setTimeout in yacysearch.html

fix for ssl-connections from proxy-usage staying in close-wait-state
+ some extra 'close' in HttpClient

added new icon/image for latest commit

- replaced 'edit' link with a clone symbol in Table_API_p since that is
what it does: it clones the crawl, it does not change the crawl.
- moved the appearance of this clone link to the type column since this
makes it visible also if the URL column is not visible.

Merge remote-tracking branch 'jensbees/crawlexpert-post'

rolling back wrong merge

reverting local changes to project.xml

check URL on inital load, if set

add a hidden field for "crawlingstart" since jQuery omits the submit button value

link recorded crawl starts back to "CrawlStartExpert_p" in "Process Scheduler"

use integer types for boolean values

correct state handling for country code restriction

correctly set initial state for load filters

set form boolean values to 0 & 1 instead of false & true

Remove some post parameters, if they are set to default values, as their values are already set by YaCy. Added some documentation.

make legacy http server (serverCore) implement YaCyHttpServer interface

- Added new solr field url_file_name_tokens_t which stores the file name
tokens. This can be used to enhance the ranking.
- Added also a rating_i field as basis for later usage.
- enhanced the tokenization process.

added the anchor text within web pages to the searcheable entities of a
web page. This can be of benefit for the ranking if these fields are
used for boosts.

list more fields available for search and for ranking boosts

fix false fill NAME_CACHE_MISS-DNS-Cache in case of a timeout
for eg. caused by massive requests when crawl from file

Use list template for "reloadIfOlderNumber".

Dynamic toggling of form fields, based on passed in and selected values. This will also cut down the post string by disabling not needed fields.

removed unused initialization method

removed warnings

adding synonyms from German OpenThesaurus ready for use in YaCy

try to guess default crawling mode, if none set

assign strings before checking contents

set default values for string parameters

made crawStartExpert_p aware of post variables; extended template where needed

Patch the citation index for links with canonical tags.
This shall fulfill the following requirement:
If a document A links to B and B contains a 'canonical C', then the
citation rank computation shall consider that A links to C and B does
not link to C.
To do so, we first must collect all canonical links, find all references
to them, get the anchor list of the documents and patch the citation
reference of these links.

add call to AccessTracker to jetty security handler

adjust YaCySecurityHandler to Jetty 9 conventions
- mainly adjust prepareConstraintInfo to use the RoleInfo.setChecked as in Jetty Source distribution
- use constraint check behavior as in ConstraintSecurityHandler
  see http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/jetty-security/src/main/java/org/eclipse/jetty/security/ConstraintSecurityHandler.java?id=jetty-9.0.5.v20130813

- expand localHostName check of AbstractRemoteHandler
    to pevent request is handled as proxy request 
- make domain handler not relay on included path in resolved .yacy address

upd  build script from rc1/master

update Jetty server init() to activate yacy-domain and transparent proxy handler
- adding  domain & proxy handler to a context (as it was in inital design)
     (context required for dispatcher)
- make handler context and servlet context parallel available 
     (to allow use of YaCyDefaultServlet to handle legacyServlets)
- set transparent proxy request handled after dispatch.forward to skip further handling for .yacy domain requests

- add welcome file (index.html) support to YaCyDefaultServlet
- change SolrServlet default search field (&df) to text_t

added two more fields source_cr_host_norm_i,target_cr_host_norm_i in
webgraph and an addition to postprocessing to copy all cr ranking
attributes to the link edges associated to the postprocessing documents

added parsing of http-canonical tags (untested, could not find an
example page)

do not fail if a wrong boost key is used; instead, print only a warning
See also: http://bugs.yacy.net/view.php?id=293

add default field (&df) to SolrServlet query if missing 

- crawl profiles shall be deleted from active and passive stacks if they
are deleted to terminate the crawl because otherwise the crawl will go
on after the load-from-passive stack policy.
- better check if a crawl is terminated using the loader queue.

hack to remove most of the warning: deprecated messages (but not all,
one is left)

reverted RemoteInstance to deprecated methods of httpClient-4.2
this should work with current remote-Solr-Instances

some dns-timing changes:
since httpclient uses the domain-cache it is useful not to clean the
domain cache until crawling is running (domains are filled into this
cache)
On huge crawl-starts (eg. from file) my DNS did not follow the high
rates - so I reduced the rate and give some more time(-out)

when profiles are cleaned, there shall be first a callback showing which
profiles are cleaned. This shall enable a profile-termination-driven
postprocessing. To do this, index writings must carry the profile key
which will be implemented in another (next) step.

removed superfluous class

Added new data structure to be used by the balancer (not used yet).
These data structures will enable the balancer to store the crawl queue
into individual queues, one each for a single host.

refactoring

added location navigator which causes that the image to the map search
is visible whenever a location is available in the search result.
To activate this, the search.navigation property in yacy.conf must be
modified to the new default values.

automatically removed Places autotagging if no location library is
wanted

- added hosthash to all documents (also fail documents which is needed
there for deletion), this fixes a problem for the deletion of old
documents for new crawl starts
- added clickdepth and citation computation for fail documents

close include byte buffer after usage

remove double declaration of TLD_any_zone_filter

- reduce Jetty debug logging
- fix Context path initialization

adjust to DigestURI changes from master to DigestURL

use more YaCy HeaderFramework constants (instead of Jetty's)

implement a YaCyDefaultServlet to handle YaCy-servlets within Jetty server
- the implementation is inspired by Jetty's DefaultServlet
- handles static html content and YaCy servlets
- translates between standard servlet request/response and YaCy request/response specification
With the implementation of YaCy-servlets as servlet instead via a jetty handler it's closer to servlet standard and carries less jetty specific dependencies.

hack to add all tokens from the url to text_t. This was working for the
RWI index (and still is working) but not for solr-only search indexes.
Maybe we should find a solution using a separate search field instead.

do not check all words with synonym library if the library is empty

disabled ipv4 preference to enable ipv6-only networks like freifunk

CPU problem when pusing to the error cache; wrong class,
ConcurrentHashMap needed for concurrency

better and consistent deletions for error urls

set anchor rel attribute of all links to "nofollow" if the html meta
contains a robots:nofollow or if the http header contains a
"X-Robots-Tag: nofollow"

add Solr Servlet

Merge remote-tracking branch 'origin/master' into jetty

update Info.plist for Jetty 9 jars

remove redundant declaration of USER_AGENT

add mandatory p2p parameter to templatePattern

adjust setHandled to request of call parameter

remove references to 10_http thread (legacy http server)
and add needed get/set function to jetty http server wrapper

decreased the responseHeaderDB database which is now flushed more
frequently. This will preserve more documents in the cache in case of a
crash.

added option for daterange properties in GSA interface to use an left-
or right-open date range;
i.e. using daterange=..2013-09-09 or daterange=2013-09-02.. additional
to daterange=2013-09-02..2013-09-09

- remove double initialization of jetty
- refactor some var assignments

- delete not needed old jetty jars from libt
- add jetty to Info.plist

adjust test/ParserTest to dc_description data type

added tools to visualize image generation (i.e. during testing)

changes to adjust jetty to recent code changes

forced deletion of ZURL entries for a specific host for each host that
appears in the crawl url list

refactoring (im preparation for new removeHost method)

fix for a problem which ocurres when a site is crawled where the start
url is redirected.

turned images_alt_txt back to images_alt_sxt because it is not necessary
to index the alt text. Indexed image Text is in images_text_t

activated image type navigation for image search

added an option to exclude image search results from text search. This
is on by default.

added an option to put image links to the crawl queue and handle these
like normal documents. Using this option (by default on at this moment;
this might change soon) it is possible to get the exif data into the
search index to be used in image search.

avoid to create unnecessary objects

moving image description text to image text field

remove all_words, all_strings lists from QueryGoal
- only used for text highlighting in parser text (ViewFile.html) which can be done with include_strings only

optimized QueryGoal (reducing mem and computation by removing all_hashes)
- all_hashes used for text highlighting and word distance computation which can be done with include_hashes only

link strings should not be tokenized

image alt tag strings should be tokenized

next development release step to reflect the extension of the solr api
with javabin format capability

added special handling of the BinaryResponseWriter in the solr interface
which makes it possible to use solrj with the javabin format which is
much better (compressed, no xml overhead, java object streams) and
faster. Furthermore, this enables the 'shards' option in the solr
interface which connects one solr (YaCy) to another solr (YaCy) ad-hoc.

return field names in index instead of in schema for SolrServerConnector.getFields

index migration to lucene 4.4

fixed a crawler bug where a double-occurring url was not re-crawled
because the double-check error was written to the error-db and never
deleted. No the error-db is cleared on every start and these
double-messages are not written to the error-db any more.

removed an assert which causes a shortcut call circuit

added missing files :(

add back menu to Surftips page (currently no menu is displayed)

removed link.gif as link button because this image is now shown
automatically for expernal links

less logging for postprocessing (this was a debugging logging with high
CPU load)

revert del _blank (last commit) in template

add some href target=_blank to ext. links with external icon

- remove possible double initialization of rdfa parser
- use ordered list to use preferred parser for mime/extension first (relates to html, rdfa, argument parser)
- harmonize xhtml extension config for the 3 html base parsers

FieldReIndex: omit active vocabulary fields from reindex detection

make Vocabulary Navigator tags a hard result entry filter
by checking vocabulary tags also for rwi results (currently a filter is applied to the solr query)

TODO: as vocabularies are only locally valid, auto-switch to Searchdom.LOCAL could be considered.

improve tld: query modifier filter pattern  (to prevent tld:net  accepting  www.abcinet.org)

Field Re-Indexing: display list of fields in reindex queue
change servlet to display statistic on 1st click (instead after refresh)

clear some caches before reporting low Memory
do not break lines in Network-table-rows

Index Administration - Field Re-Indexing: exclude internal Solr _version_ field from obsolete field check

rise autoCommit maxTime to 3 Minutes to reduce IO
lower mergeFactor again (5) for less segments

add branch to packet-name if not build from master

fix / workaround for
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4750
+ Seed.hash should be final

protection against self-ping (may be cause by fraud attempts)

added donation plea to the about box as default (you can replace this in
your peer!)

more space in the about section

fix to collection schema; bug appeared for _txt fields with empty String
as content

added a stub of a class for crawler redesign

Improved external links.

*) image links will not be marked (if they have class "yacylogo" or
"forceNoExternalIcon")
*) external links in menu on left (and "fork me"-banner) will open in
new tab/window now

External links in HTML interface are marked as external with small icon.

*) added new icon
*) added CSS rules to mark all external links except search results
(target="_self")

fix for NPE which happens within solr code at MultiMapSolrParams.java,
line 52 in case that the array arr.length == 0

update to pdfbox 1.8.2

cleaned sourcepaths from eclipse classpath

next development cycle

Field Re-Indexing: prevent endless error loop in ReindexSolrBusyThread on Solr exception (by skipping query causing the exception)
(occured during testing while working on q=store:[* TO *])

htmlParser: closes input stream -> changed it to leave it open for a reset (used by AugmentParser - even if this is practically not used),
note: stream.close is done by caller (Textparser.parseSource)
- removed unnecessary reset in AugmentParser
- added stream.mark in tdfatripleimpl. to make stream.reset work here

reverted start script options - yacy on windows did not start with the
given values

emergency fix for crawler: this will otherwise cause loss of complete
crawl queue if latency of remote system is too low