YaCy Release 1.90

Merge remote-tracking branch 'origin/master'

Conflicts:
	htroot/Load_MediawikiWiki.java
	htroot/Load_PHPBB3.java
	htroot/ViewImage.java

Merge remote-tracking branch 'origin/master' into docker

Merge remote-tracking branch 'origin/master' into docker

Merge remote-tracking branch 'origin/master' into docker

Merge remote-tracking branch 'origin/master' into LibreJS

Aplied GNU licensing recommendations.

Added license information for YaCy owned js files

Merge remote-tracking branch 'origin/master' into docker

introduce a translation edit servlet Translator_p.html YaCy's UI text translation
This is the 1st rudimentary approach to support the translatio utilities.
It allows currently to edit untranslated text and save it in a local translation file
in the DATA/LOCALE directory.
+ refactor Translator (less static's) to leverage on class overrides and support garbage collection for this 1 time routine
+ adjust TranslatorXliff to check for local translations in DATA/LOCALE,
  this includes storing manually downloaded translation files in DATA as well 
  (to keep default untouched)
+ on 1st call of Translator_p a master tanslation file is generated, checking
the supported languages for missing translation text (later this masterfile is planned to part of the distribution, to harmonize translation key text between the languages)
Outlook: the local modifications (possibly as translation fragments instead of complete file) to be shared with maintainer using xlif features. 

upd to Jetty v9.2.17.v20160517

Merge remote-tracking branch 'origin/master' into docker

upd to Solr 5.5.1

Merge remote-tracking branch 'origin/master' into docker

upd to Jetty v9.2.16.v20160414
- exclude unused mime4j
- remove unused yacy-cora build

manually pulled - now pushed

keep and update existing xlf translation master instead of create new
in utility CreateTranslationMasters
+ small fixes in lng's

keep ConfigLanguage_p as 1st entry in exported translation file 
+ rem untranslated text & some typo fixes in several translations
(considering to create a translation master file to harmonize entries)

Introduce a TranslatorXliff wich can read/write xliff from/to internal translation map.
This eases up suggested initatives from http://mantis.tokeek.de/view.php?id=649
Allows longer term also to store translation maps for the htroot files 
in standardized/reuseable xliff format ( http://docs.oasis-open.org/xliff/xliff-core/xliff-core.html ).
+ added test case creating and comparing xliff file with internal custom prop file.
(currently the introduced class is not used in core code)

load handler for Transparent Proxy on startup only if feature is activated
to save the resources and keep handler chain small if the feature is not used.
+add a warning message on settingsack_p page to restart on first activation

Merge pull request #4 from yacy/master

Get my fork synced #4

upd to Solr 5.5.0

fix sevenZip parser exception with LZMA BCJ2 format archives,
see http://mantis.tokeek.de/view.php?id=641
+ sanitize some type cast's in modified sevenZip lib

moving the J7Zip-modified source and Maven build to libbuild
from main pom. 
Using source included in j7zip-modified.jar.
This combines all external lib preparation in the libbuild main pom.

upd to Jetty 9.2.15.v20160210

result heuristic (also used in greedy learning mode) to use outbound links if result is full index doc. Otherwise use default loader methode.
- Above brought up that parser start url parameter, declared as AnchorURL uses only methodes of parent object DigestURL (changed parameter declaration accordingly).

Created ViewFavicon class specialized in favicon viewing.

Main image processing is now in ImageViewer, used by both ViewImage and
ViewFavicon.

Fixed URIMetadataNode.getFavicon to use non-standard icons with no size
ass fallback.

upd to Solr 5.4.1

Improved HTML icons indexing and rendering in search results.

See http://mantis.tokeek.de/view.php?id=629

Merge pull request #3 from yacy/master

Get my fork synced #3

upd to pdfbox 1.8.11

Merge pull request #2 from yacy/master

Get my fork synced #2

add support of compressed swf to swfParser
from JavaSWF2 (source compatible to WebCat).
Moved swf file signature check to parser
Changed use of synced vector to list swf InStream

import WebCat swf parser custom source package
This package is not available as jar (used jar is a custom compile as we 
use just a portion of the package) 
WebCat package is not maintained. To be able to fix bugs, source extract 
of swf parser imported here.

move test classes to test/java (subdirectory as in Maven standard subdir layout)
because ViewImage*Test.java breaks test run

Merge pull request #38 from luccioman/master

Refactoring : use StandardCharsets instead of hardcoded charset names

update on translation files
- delete removed servlets
 AugmentedBrowsingFilters_p.html (de)
 CrawlStartIntranet_p.html (de)
 IndexCreateWWW***Queue_p.html (de)
 Ranking_p.html (de)
- add
 IndexCreateQueues_p.html
- rename
 Settings_Http.inc -> Settings_ProxyAccess.inc
 Language_p.html -> ConfigLanguage_p.html

Refactoring : use StandardCharsets constants instead of hard-coded
charset names.

bump to solr/lucene 5.4.0

Merge remote-tracking branch 'choose_remote_name/master'

apply default css styles (class btn) to submit buttons 

upd to ImageIO 3.2.1

apply more default css styles (class btn) to submit buttons

upd to Jetty v9.2.14.v20151106

BMP and ICO image formats support : integrated /haraldk/TwelveMonkeys
imageio-bmp-3.2 library.

 - better BMP format flavours support
 - handle PNG encoded icons
 - handle transparency
 
Added some javadoc url references to .classpath

avoid File.deleteOnExit() on temp files
JVM registers each file in a list regardless of already deleted and never
cleans up the list during runtime.
This accumulates to a considerable amount of mem during large crawls and/or
long uptime.
To tackle this, all temp files are now created in a subdir of java.io.tmpdir 
and the jvm tmpdir property is set to this subdir, which is deleted by
code on shutdown.
Additionally let pdfParser use this tmp subdir too.

upd to TwelveMonkeys ImageIO 3.2

Refactoring : default favicon and image processing errors.
 - moved default favicon processing from ViewImage to
yacysearchitem.html : when previewing ico image search results we don't
want a default favicon be displayed
 - throw an IOException ending in a HTTP 500 error when image processing
fails, rather than returning a null result : behavior is more consistent
accross browsers (for exempla Chrome and Firefox), especially with new
default favicon display system

differentiate api call getLocalPort() from getConfigInt()

upd to jetty 9.2.13.v20150730

upd httpclient-4.5.1, httpmime-4.5.1, httpcore-4.4.3, commons-compress-1.10

implement ajax crawling scheme for ajax sites which adhere to the proposed use of hash-bangs to provide html content
see freshly deprecated https://developers.google.com/webmasters/ajax-crawling/
Implementation improves parsing of the homepage (ajax page) which uses metatag "fragment" in header and parses supplied html snapshot instead of mostly empty ajax/scripted page.
Implementation supports also hash-bang urls (url with anchor starting with ! like  ...path#!hashfragment) but our crawler filters it
(use of hash-bang is controversly discussed and proposal is deprecated, makes no sense to adjust the crawler, but as long as it is used by some sites the minor change/improvement in htmlparser is good for some time).
Quick - how does it work
- if metatag fragment with content "!" is found
   - htmlparser tries to get content of htmls snapshot (using a different url)
   - htmlparser returns 2 documents (original url and snapshot content - but using same original url)
- after parsing result documents are joined (and stored to index containing content also from snapshot page... as the original ajax page contains typically no parseable html content)

Integrated haraldk/TwelveMonkeys library to first add TIF image format
support.

upd to solr/lucene 5.3.1

Merge pull request #14 from luccioman/master

Translator refactoring : no more regular expression processing

Translator refactoring : to simplify locale files writing, process keys
as simple string and no more as regular expressions.
Updated all locale files to adapt to refectored Translator : removed
useless escaped characters and did minor corrections.
Performed minor syntax corrections on some html source files.
Added an util to translate all html source files with all locales
without launching full YaCy application.
Corrected main arguments parsing on other translation utils.

when many crawl queues are generated, this NPE can occur; probably
caused as concurrency issue:
W 2015/09/05 14:09:10 ConcurrentLog java.lang.NullPointerException
java.lang.NullPointerException
	at java.util.TreeMap.rotateRight(TreeMap.java:2239)
	at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2271)
	at java.util.TreeMap.put(TreeMap.java:582)
	at net.yacy.kelondro.table.Table.<init>(Table.java:235)
	at net.yacy.crawler.HostQueue.openStack(HostQueue.java:229)
	at net.yacy.crawler.HostQueue.getStack(HostQueue.java:204)
	at net.yacy.crawler.HostQueue.push(HostQueue.java:397)
	at net.yacy.crawler.HostBalancer.push(HostBalancer.java:237)
	at net.yacy.crawler.data.NoticedURL.push(NoticedURL.java:184)
	at net.yacy.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:355)
	at net.yacy.crawler.CrawlStacker.job(CrawlStacker.java:134)
	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at
net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:101)
	at
net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:82)
	at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
	

fight the fieldcache by usind DocValues: in Solr-5.x the fieldcache has
moved and was not cleared anymore. This results in an huge fieldcache.
(http://lucene.apache.org/#highlights-of-the-lucene-release-include
https://issues.apache.org/jira/browse/LUCENE-5666)
Here I try to use DovValues where it is possible.
For this I used the Api-Scheme as new basis für the Solr-Schema.
This needs at least a complete optimization of the Solr-Index to get a
smaller FieldCache.
Everything that is indexed with these setting will not use the
Fieldcache at all.

added a new facet type based on a probabilistic classifier using
bayesian filters. This can be used to classify documents during
indexing-time using a pre-definied bayesian filter.

New wordings:
- a context is a class where different categories are possible. The
context name is equal to a facet name.
- a category is a facet type within a facet navigation. Each context
must have several categories, at least one custom name (things you want
to discover) and one with the exact name "negative".

To use this, you must do:
- for each context, you must create a directory within
DATA/CLASSIFICATION with the name of the context (the facet name)
- within each context directory, you must create text files with one
document each per line for every categroy. One of these categories MUST
have the name 'negative.txt'.

Then, each new document is classified to match within one of the given
categories for each context.

removed warnings

a collection of search query enhancements:
- fixed superfluous space in query field list
- fixed filter query logic
- removed look-ahead query which caused that each new search page
submitted two solr queries
- fixed random solr result orders in case that the solr score was equal:
this was then re-ordered by YaCy using the document hash which came from
the solr object and that appeared to be random. Now the hash of the url
is used and the score is additionally modified by the url length to
prevent that this particular case appears at all.

upd to pdfbox-1.8.10

upd to Solr-5.2.1

move java environment parameter setting disabling SNI (Server Name Indicator) support for https connections from code to startup script allowing admin to ~easy/transparent alter the YaCy default FALSE setting.
Background: some user report problem with connecting/crawling some sites via https which require SNI support (by default switched off in YaCy). On the other hand systems not demanding SNI support are sometimes not properly configured and due to a bug/feature in java 1.7 connection is aborted. The later is more often the case, so the default is still fine. With the java start parameter expert user can no alter the startparameter to -Djsse.enableSNIExtension=true (java default) if they crawl more hosts requiring SNI support.
The alternative to let YaCy try both during https handshake (deep inside the httpclient) is not pursut at this time.

Use language-detection library for increased accuracy

migration to Solr 5.2: huge benefits - this is a lot faster!

This is a very complex migration: many classes had been renamed or
removed, dependencies changed and the solr index type is now aligned to
be a solr cloud repository.
Together with the Solr 5.2 library update, one other dependent library
had been updated as well: httpclient 4.4->4.4.1

Older indexes are migrated from 4_10 to 5_2. However, the new index
structure is more efficient and we recommend to re-index everything.
Please use the index export before you do the update to a large
surrogate xml file. After the update, start with an empty index and then
initialize this with your dump.

Update to Jetty-9.2.11 - a bugfix-release that did not solve my
Problems, but does not harm anything

added intensity option to graphics

added a full solr export to the IndexControlURLs_p.html servlet. The
export function is also now the default export option. The export file
format for a full solr export is very similar to a solr search result
xml, only the <lst name="responseHeader"> tag is missing.

The exported xml has a special line termination feature: all documents
will be exported into a single line without any CR in between. That
means that every document is completely inside a single line. While this
is not readable at all for humans, it is very useful for linux line
processing scripts, like grep. Using grep it will be easy to select
single documents which match for a given pattern.

Such dumps shall be importable with the DATA/SURROGATE/in import
function, but that import is not yet adopted to the new file format.

Init remote crawler on demand
If remote crawl option is not activated, skip init of remoteCrawlJob to save the resources of queue and ideling thread.
Deploy of the remoteCrawlJob deferred on activation of the option.

refactor getBookmark 
to consistenly check existance by != null (w/o throwing exception on not found)

update to bootstrap.css 3.3.4

enhanced timezone managament for indexed data:
to support the new time parser and search functions in YaCy a high
precision detection of date and time on the day is necessary. That
requires that the time zone of the document content and the time zone of
the user, doing a search, is detected. The time zone of the search
request is done automatically using the browsers time zone offset which
is delivered to the search request automatically and invisible to the
user. The time zone for the content of web pages cannot be detected
automatically and must be an attribute of crawl starts. The advanced
crawl start now provides an input field to set the time zone in minutes
as an offset number. All parsers must get a time zone offset passed, so
this required the change of the parser java api. A lot of other changes
had been made which corrects the wrong handling of dates in YaCy which
was to add a correction based on the time zone of the server. Now no
correction is added and all dates in YaCy are UTC/GMT time zone, a
normalized time zone for all peers.

upd to PDFBox 1.8.9

upd to Jetty 9.2.10

Show dates in the content of a document in the search result:
- if an eventDate is given in the search result, replace the document
date with the event date and prefix it with the string "on ".
- the document date is omitted if a date from the cent is shown

Added also the date as fields in the json and rss result sets.

added a new way of content browsing in search results:
- date navigation

The date is taken from the CONTENT of the documents / web pages, NOT
from a date submitted in the context of metadata (i.e. http header or
html head form). This makes it possible to search for documents in the
future, i.e. when documents contain event descriptions for future
events.

The date is written to an index field which is now enabled by default.
All documents are scanned for contained date mentions.
To visualize the dates for a specific search results, a histogram
showing the number of documents for each day is displayed. To render
these histograms the morris.js library is used. Morris.js requires also
raphael.js which is now also integrated in YaCy.

The histogram is now also displayed in the index browser by default.

To select a specific range from a search result, the following modifiers
had been introduced:
from:<date>
to:<date>
These modifiers can be used separately (i.e. only 'from' or only 'to')
to describe an open interval or combined to have a closed interval. Both
dates are inclusive. To select a specific single date only, use the
'to:' - modifier.

The histogram shows blue and green lines; the green lines denot weekend
days (saturday and sunday).

Clicking on bars in the histogram has the following reaction:
1st click: add a from:<date> modifier for the date of the bar
2nd click: add a to:<date> modifier for the date of the bar
3rd click: remove from and date modifier and set a on:<date> for the bar
When the on:<date> modifier is used, the histogram shows an unlimited
time period. This makes it possible to click again (4th click) which is
then interpreted as a 1st click again (sets a from modifier).

The display feature is NOT switched on by default; to switch it on use
the /ConfigSearchPage_p.html servlet.

Fix for Jetty "JetLeak" bug: update to jetty 9.2.9
The bug was inside the jetty library, for details see:
http://blog.gdssecurity.com/labs/2015/2/25/jetleak-vulnerability-remote-leakage-of-shared-buffers-in-je.html
We recommend to update your YaCy peer with this bugfix.

Changes to improve compatibility with OpenBSD. (see
http://forum.yacy-websuche.de/viewtopic.php?f=8&t=5503)

upd to Jetty 9.2.7

added a html field scraper which reads text from html entities of a
given css class and extends a given vocabulary with a term consisting
with the text content of the html class tag. Additionally, the term is
included into the semantic facet of the document. This allows the
creation of faceted search to documents without the pre-creation of
vocabularies; instead, the vocabulary is created on-the-fly, possibly
for use in other crawls. If any of the term scraping for a specific
vocabulary is successful on a document, this vocabulary is excluded for
auto-annotation on the page.

To use this feature, do the following:
- create a vocabulary on /Vocabulary_p.html (if not existent)
- in /CrawlStartExpert.html you will now see the vocabularies as column
in a table. The second column provides text fields where you can name
the class of html entities where the literal of the corresponding
vocabulary shall be scraped out
- when doing a search, you will see the content of the scraped fields in
a navigation facet for the given vocabulary

replace with CommonPattern.SPACE for split

applying precompiled CommonPattern.COMMA.split to all places where
split(",") was used

replaced old JavaApplicationStub for Mac Application framework with new
script. Adopted the YaCyApp environment and fixed a problem in the
startYACY.sh application wrapper which caused wrong usage of logging
option -l which caused that files had been written to the YaCy
application folder.
As a result of this fix, it is not necessary any more to change path
settings in Info.plist if libraries are changed.

Fixed rendering for many mobile devices.

JavaScript License fixes for LibreJS compatibility

fix MessageBoard test for null data

Fixed license link for display on DockerHub

Fixed line end causing build error.

Fixed link to docker readme (mediawiki format)

Fixed as background process title

fix MessageSend servlet response if called for passive peer

fix SeedDB.get(byte[]) hash string compare (for returning own seed shortcut)

fix NPE on missing crawldepth_i

fix NPE in citation servlet on empty text field

Fixed initial admin password initialization

fix migration, account for ssl port in config (for auto-disable https)

fix logger name

fix rwi doubledom return best (highest) ranking

fix some line breaks and mark additional lines w/o translation

fix line break errors in translation files

fix some format errors in language files

fix maven packaging (include langdetect directory)

fix use of not initialized variable (m_LiteralDecoder)

fix SolrSelectServlet authentication to  default to true

Fixed json search results from commit "Improved URLLicence reliability"

fix json date facet displayname

fix for connections not getting closed to get favicon.ico during seach

fix for 'move test classes to test/java'

fix typo in Steering.html

fix NPE on empty blog importfile parameter

just remove debug leftover

fix reading of ippattern config array in URLProxy

fix (todo) remove redundant holding of email link nameproperty in parser document

fix dc:date in mediawikiimporter/document.writexml to use lastmodified

Added useful debug JVM parameters as comment.

fix in error handling

fix for npe

prevent exception on repeated ViewImage with same urlLicense

fixed typos

fixed classpath

Corrected IcedTea version. See http://mantis.tokeek.de/view.php?id=615

fix link to logo (yacysearch.xsl)

Ensure closing of InputStream even when an exception occurs.

fix detection of https port changed after set in System Admin

fix typo in WikiCode coordinate calculation

fix for image size field values (must be multi-valued)

fix link target on iframe list in CrawlProfileEditor

fix for latest merge

fix test methode (add throw for URIMetadataNode)

fix init of error cache, use latest faildates => load_date_dt

avoid runtime exception by earlier testing for seed.ip=null

fix for filesystem crawl

revert of fq transformation (recent fix)

more fixes for special windows paths

patch for bad windows file paths

fix for index import

fix for unresolved pattern

fix for division by zero

disabled debug thread dumps

fix NPE in addToIndex when used outside searchEvent

added temporary debug output in http client

precaution against NPE on createorgetBookmark on search result

fix typecast for css links

fix for filter queries

fix for not valid json in case that topics are switched off

logging fix

better debugging of fq

fix NPE in location search on missing/empty PubDate in underlaying rss data

shorten exception loggin on not available connection in  Load_RSS_p servlet

fix NPE on viewfile of url not in index

fix: banner did not show link and qph for portal mode

fix get fresh_date_dt  to allow returned value to be date in future

fix MultiProtocolURL mailto protocol detection

fix link to DeReWo project page

fix link to DeReWo download file

bugfix for fixed host/port

fix NPE in snippet computation

fixed httpclient lib paths in ant build

npe fix for latest scraper feature

fix: searchoption hint for heuristic

IPv6 Fix for push interface

fix for failed selection of terms in faceted search with vocabularies

fix for wkhtmltopdf (custom header does not work)

fix for urlmaskfilter

fix for vocabulary on/off setting

fix for shell script

release 1.90

another peer list update

small refactoring + do not accept too old peers during bootstrap

test case for HostBalancer issue in intranet mode
with file:// protocol, 2 hostqueues accessing same cache file concurrently
http://mantis.tokeek.de/view.php?id=668
Reason seems to be diff. hosthash key of hostqueues on reopen. 
Internal queue key and external representation (directoryname currently hostname.port) must be adjusted to fix it (not done yet).

added another principal peer

Revert on commit related to another personnal branch.

Added instructions for log control and upgrade

remove wrong log line in CrawlSwitchboard
+ don't allow CrawlSwitchboard to exit application
making network param unused

prevent dealing with -UNRESOLVED_PATTERN- eventID parameter in html includes
on first landing on search page

add de hint translation for CrawlStartScanner_p
rem missing translation line in other lng

ignore empty input lines in FileUtils.getListArray() to poka joke blacklist read.
equalizes behavior with getListString()
improves: case were blacklist file contained a undesired empty line, not 
fixed by blacklist-cleaner.

rise limit of reversion to >9999

fix NPE in CrawlMonitorRemoteStart servlet due to missing startURL
+ add a startURL attribute while generating news record for above (in Crawler_p)

remove redundant/unused translation in ru.lng, unify Network.html header
+ dto. in uk.lng and comment-out some German leftovers.
+ upd master.lng accordingly

exclude rejected results from result count 
  (by using the resultcontainer.size instead of input docList.size)
skip waiting for write-search-result-to-local-index
  (by removing the Thread.join - which will bring a small performance increase)

wiki header code test string add "closing" tag

internal wiki code, require header line to start with markup 
(to allow something like  "one=two"  as text)
+ incl. test case

add maven release profile plugin version (following maven's suggestion)
+ upd some plugin version

fix missing quote in gr.lng, it.lng, add sentence in master.lng.xlf
and remove 2 not needed entries

upd to slf4j-api-1.7.21.jar with related jars

Merge pull request #58 from luccioman/responsive

Make YaCy usable on small screen devices

upd to pdfbox-2.0.2.jar

upd to httpcore-4.4.5.jar

Add a tranlation master (locales/master.lng.xlf) in xlf format.
This is used by Translator servlet to present text to be translated for 
all languages. The file was created by the entries of the 2 best maintained
languages (ru.lng and de.lng). 
+ add view translation to de.lng and harmonize some entries to master.lng.xlf

fr.lng add entry for Performance_p.html, submenuDesign.template
correct some quoted entries and equalize some keys to common

Translator servlet filename drop-down list on "filter untranslated" switch,
filter list to contain only files with not translated items

Merge remote-tracking branch 'origin/master' into responsive

Merge remote-tracking branch 'origin/master' into docker

add button (Translator servlet) to switch one already translated line in edit mode

have Translator servlet adhoc apply added translation by translating a single file
+ fix NPE in Translator, coming from translation read by TranslatorXliff 
  which allows null content for not translated key's

Merge remote-tracking branch 'origin/master' into docker

clean out not used Switchboard variables
counter indexedPages, const xstackCrawlSlots

rem German lines in sk.lng 
and line not included in file (status_p.inc)

Translator to take caution of word boundaries to identify text portion to
be translated. To avoid key="TEST" sourcetext="this is a myTESTcase for it"
translation of partial terms/words.
Add check of word boundary before and after sourcetext (incl. take care
of current praxis for key to be delimetered by > < 
+ add test case

Added last missing JS licences

upd to jsoup-1.9.2

add DE translation Translator_p
+ html conform order form/fieldset

Added license info for jquery.tagsinput.

Also updated some license urls and identifiers to ones recognized by
LibreJS.

Added remaining scripts license information

Added d3.v3.js license information

Adjust TranslatorXliff to load default 1st and merge downloaded or modified local translation.
process 1. load default from locales/*.* 
        2. load and merge(overwrite) from DATA/LOCALE/*.* (can be partial translation as it is merged)
- include all entries from DATA/LOCAL to be edited in Translator servlet
  and save just modifications (instead of full list) to DATA/LOCALE

This shall make it easy to share modifications.

fix de, uk translation RemoteCrawl_p.html, WatchWebStructure_p.html
harmonize status.html

Merge pull request #54 from luccioman/debian

Update debian dependency to allow installing with java 1.8

optimize Translator
- translateFilesRecursive: load translation once (reduce io), return true on complete success
  - remove resulting unused translateFiles() variant
- translate: use StringBuilder parameter (skip toString conversion)
- remove not needed static declaration
- upd some javadoc

disable input field of translated text in Translator_p servlet
As currently only adding missing translation is supported

Added JavaScript license information

First pass applied for YaCy index and administration first page, checked
with LibreJS 6.0.13.

fix accidental use of Java 8 function in Translator_p
thanks to luccioman  

Improve final image size by concatenating some strategic commands

on language file delete, del local editfile too (in ConfigLanguage_p servlet)

upd to commons-fileupload-1.3.2

git, compile and cleanup in one instruction

Updated docker readme

Added image variants, tags, and license information.

Improved curl commands (fail if URL is unreachable).

remove with localHostNames redundant (but unused) publicIPv4HostNames and publicIPv6HostNames
to free unused resources

Compacted some instructions to reduce final image size

Created a Dockerfile for yacy base on Alpine Linux

Updated comments

Use transparent addToCrawler in AutoSearch instead of addToIndex

This would likely also be of advantage for RSS import/schedule as
following bug-reports suggest
http://mantis.tokeek.de/view.php?id=569
http://mantis.tokeek.de/view.php?id=655

Added details about volumes management.

Set process as non-root with "USER" instruction.

Thus there is no need to get sudo package.

Added virtual machines and containers block

Use latest official Docker java image as a basis.

This image is a debian Jessie with an openjdk-8 from debian backports.
This avoid downloading openjdk dependencies through apt-get each time
YaCy docker image is built.

Run yacy daemon as non-root for improved security.

Added curl dependency to allow running stopYACY.sh and /bin sh scripts.

correct translation key "About This ...." for simpleheader.template
in de, ja
+ rem obsolete key "Peer Owner Profile" in cn, sk, uk for simpleheader.template
+ hide empty account lines in ViewProfile servlet

fix 2 more servlet RuntimeException in intranet mode thrown due to seed.getIP()
returning null in intranet mode (in servlets: ConfigSearchBox, Load_PHPBB3
+remove unused (const ∅) seed.IPTYPE

fix: assure close inputstream in TranslatorXliff after reading xlf file
by using try-wiht-resource block

fix RuntimeException in ConfigRobotsTxt_p in intranet mode
due to getIP returning null, by removing redundant property setting of "clientname"
which is globally set by template engine

Set more generic java runtime dependencies.

This is consistent with official Debian java policy :
https://www.debian.org/doc/packaging-manuals/java-policy/x105.html

fix NPE in Load_MediawikiWiki servlet in intranet mode
  - in intranet mode getip returns null causing a NPE
  - adjust starturl (which was set to http://localip/repository) which is never the start url for the Mediawiki
+ correct javadoc for seed.getIP()

Enable yacy install on debian with openjdk-8 only

Also added curl or wget in Suggests as they are required to run sh
scripts in /bin.

added hint in status to upgrade to java 1.8

added column in network table for JRE

add the JRE version in the seed. Purpose: identify if it is possible to
migrate to new JRE version

Merge pull request #52 from luccioman/docker

Docker image building from latest sources

Merge pull request #53 from luccioman/accessibility

Use aria role to reduce rssTerminal verbosity.

add InputStream close after reading input file (Vocabulary_p servlet)

Use aria role to reduce rssTerminal verbosity.

This is a fix for http://mantis.tokeek.de/view.php?id=643

upd to icu4j-57_1

Merge remote-tracking branch 'origin/master' into docker

Detailed how default passord is encoded

adjust Solr select servlet to lates bugfix for boostquery (bq param)
to split query into multiple parameter on line separator in input query.
e.g. split "crawldepth_i_0^10.0 \n crawldepth_i:1^5.0"
but allow   "url_file_ext_s:jpg OR url_file_ext_s:png"  to be unsplitted

Merge pull request #51 from JeremyRand/multiple-boost-query

Fix multiple boost queries

Merge remote-tracking branch 'origin/master' into docker

make use and activate autodetect charset in Vocabulary input from file
+ revert mistake of empty cn.lng

Merge remote-tracking branch 'origin/master' into docker

Properly support multiple Boost Queries.  (Previous code was broken because it concatenated multiple Boost Queries together rather than passing Solr an array.)

Refactor escaping in config file read/write code.  Now it uses Apache Commons StringUtils instead of RegEx.

upd to PDFBox 2.0.1

Enabled autoredeploy, removed useless volume

- Autoredeploy allow dockercloud to redeploy container when a new image
is pushed on dockerhub. 
- volume /opt/yacy_search_server/DATA is already declared in docker
image and allow persistence of data inside deployed container.

del cytag trail for own index.html (save resource not used by default)

Added one-click docker cloud deploy

Improved titles levels

Created a docker cloud stack file

May be used for one click deploy on docker cloud (see https://docs.docker.com/docker-cloud/apps/deploy-to-cloud-btn/)

extend snapshot Html2Image.pdf2image to use PDFBox image export capability
if no external tool installed (and for Win)
Resulting jpg are not always perfect (if graphic included) but imho sufficient.

Merge pull request #2 from yacy/master

Merge master

Try reducing even more final docker image size

Reduce docker image size

Added docker readme

Initialize admin password from default or env

save resource for unused blacklistFiles map (#1)

Moved again Dockerfile to docker subdirectory

Moved again Dockerfile to docker subdirectory

Modified sources getting

save resource for unused blacklistFiles map

Try for automated build compatibility on dockerhub

Added Dockerfile on master

Try for automated build compatibility on dockerhub

Remove (unnecessary?) clean in ant command

Trace also hidden files in sources

Moved Dockerfile to project root

This is a try for automated build compatibility with Docker Hub

Moved Dockerfile to project root

This is a try for automated build compatibility on Docker Hub

Try for automated build compatibility on dockerhub

remove unused wget package dependency

Corrected COPY instruction

Try modifications for automated build on dockerhub

Added a Dockerfile

The goal is to automate building of yacy docker image from latest
commits.

Create .gitignore

Delete .gitignore

Update .gitignore

Update .gitignore

Delete YaCy ottdist [Builder].launch

Merge remote-tracking branch 'origin/master'

again

optimized for small y values

Update .project

Merge pull request #5 from yacy/master

Get my fork synced #5

optimize surftip data load,
double load (index, loader) not neccessary, getMetadata already suficient
+ lng file adjustments

cleanup connectPeer & yacyVersion.latestRelease usage
obsolete since 
https://github.com/yacy/yacy_search_server/commit/527b3decdee2951910bef96f5e5e915fe7c9c73e

cleanup hack for partial Solr update on multivalued datefields
has been fixed in Solr http://issues.apache.org/jira/browse/SOLR-8050

populate translation "index browser"

clean-out unused SwitchboardConstants

delete obsolete SEARCHRESULT busythread constants
not used since 29.05.2013 18:27:27
0c1a018bbde9c9e67bc000b6a3fd8dbd1706a6f3

Make sure "leftscale" is positive

apply bugfix for ChartPlotter from Pullreq 42
https://github.com/yacy/yacy_search_server/pull/42
thanks to otteresk (https://github.com/otteresk)

upd to commons-io-2.5

move obsolete submenuCookie.template translation to submenuAccessTracker.template entry

prevent "unchecked conversion" compiler message
+ include "translate" property in xlf "trans-unit" export

skip caching of knownURL during transferRWI
(cache values not used)

upd to jsoup-1.9.1

upd to metadata-extractor-2.9.1

exclude unused protocol param "duetime"
(receiver interpretes param "time" only)

keep addon/Notepad++ keyword marker on lng export 
(length of remarks devider line)
+ harmonize status_p.inc lng text

take out obsolete "ranking" directory from maven packaging

fix version in locale version file
(it's compared to full version)

upd commons-compress 1.11

delete crawl queue on init exception 
(happens occasionally on path name vaiolation and will never get resolved)

revert call condition f. migration.installSkins 
(a bug introduced in fb8ae14b217cb247d97830ff5237c80a4fe3c93e , 
see comment on that commit )

remove never used Domains.myHostNames set

translate ServerScannerList.html

support xliff for downloaded of language file

Added utility class CreateTranslationMasters to create a language independant
translation master as source to harmonize individual translation files
Included a main to create masters in YaCy an xliff format for testing 

+ restrict TranslatorXliff to use only entries with State=translated

P.S. used https://open-language-tools.java.net/editor/about-xliff-editor.html to
experiement with xlf output (haven't a Pootle avail.)

rem/mark German leftovers in translation of other languages files

clean-up submenuUseCaseAccount.template translation

fix empty drop down list in ConfigLanguage after wrong/empty download
+ add xliff translated attribut
+ append japanese lng name

translate Network History screen -> de
+ remove leftover debug line

remove unused translator routine which also doesn't handle rel path input
+ correct some language file match issues

use UTF8 to write imported language files

upd obsolete translation (blekko)

add test case for optimized toTokens()

adjust build class path (- htroot)
according to comment on https://github.com/yacy/yacy_search_server/commit/0dcda3809ed4cbc7f8a95e95f48814c24c4ed3ff

adjust utility ListNonTranslatedFiles for path compare on windows 
(backslash replace)

update de translation ConfigBasic.html (config router +upnp)

keep svnRevision in .init for convert of .conf until release >1.83

include Version config string in yacy.init (replacing svnRevision)

make migration version safe

Merge pull request #46 from reelsense/patch-1

Grammar fix - ConfigRobotsTxt_p.html

Distilled sentence - ConfigNetwork_p

I distilled the first "Distributed Computing Network for Domain" description sentence and corrected grammar.

Grammar fix

The sentence sounded odd.

Added YaCy icons

upd to poi-3.14

reduce logging (EmbeddedSolrConnector.query)
mainly to reduce the frequent metadat checks like
> EmbeddedSolrConnector.query QUERY: q={!cache=false raw f=id}xXxXxX&rows=1&start=0&fl=id,load_date_dt
(p.s. direct servlet queries logged via AccessTracker.addToDump)

move example code SearchClient out of yacycore package
to example directory

adding Mavens output dir to gitignore

upd Eclipse classpath for slf4j jars

correct "SLF4J: Defaulting to no-operation (NOP) logger implementation"
introduced earlier https://github.com/yacy/yacy_search_server/commit/d9036bb9b77da1dda0f29df6207eb529f7e3361a

activate hack in libbuild/pom.xml to include main project as build module
This allows to use libbuild/pom as main maven project to build all needed
libs to compile and run YaCy, including YaCy itself.

Furthermore, maven is well supported by Netbeans IDE so we can discontinue
to maintaine Netbeans IDE specific project settings (in directory nbproject,
which lack config for debug anyway, nbproject can be deleted and NB will
compile YaCy via maven).

Additionally adjusted pom's project name to match the name in the repository.

upd to slf4j-1.7.18
remove direct declaration of slf4j-api in pom (let maven do it's job)

del outdated lib/dependencies.txt
as alternative you may use maven build site:site goal which produces
a complete list of direct and transitive dependencies.

upd to httpclient-4.5.2

put settings_p servlet (back) as start page for System Administration
to display the options available (as tables_p only used for indepth edit)
see http://mantis.tokeek.de/view.php?id=460

0N - basic dump upload servlet infrastructure, to share index dumps
within an experimental new sharing model

harmonization of http post of files for one and several files - this had
been differently - and wrong for several files. also: base64-encoding
for gzipped push files because our data structures currently only
supports ASCII POST pushes..

0n - added experimental zeronet network which supports intranet peers
(still needs work)

0n: modified the p2p bootstraping process - rules had been too tight and
did not support the re-start of a network with just one principal peer.

update to seed list

remove unused proxyCookieWhiteList from yacy.init

more fixes for sevenZip parser exception on specific archives
Found 2 more cases were modified code throws exceptions while original J7Zip
unpacks it.
Reimplemented error prone CoderMixer2ST from unmodified J7Zip.jar. 

remove transitive dependency guava from pom
(let Maven do it's job)

fix delete of temp file after odt % ooxml parser
Close zipfile after parsing

use supplied url port to get robots.txt in crawlers hostqueue

adjust start/stop classpath in build script
(with servlet classloader no need for htroot in system classpath)

harmonize classpath with startYaCy.bat
(with servlet classloader no need for htroot in system classpath)

harmonize classpath with startYaCy.bat

fix mixed logfactory in UrlProxyServlet,
Class doesn't use functions of declared ancestor, change to extend on httpservlet

upd to slf4j-1.7.16

0N - added recording of dump exports if given time frame is not negative

0N - added option to generate index export files for a specific number
of minutes in the past and reverted latest change. The export file dump
will now contain four data elements: f - first date of index entry write
date, l - last date of index write date, n - now-date of index dump
time, c - count of numbers inside the dump. '0N' denotes a series of
changes which will lead to the opportunity to exchange index data dumps
in a way that is needed to integrate ZeroNet index data. This will be
based on index dump sharing; that causes this commit.

making WebStructurePicture_p less vulnerable to faulty host input parameter (like host1,,host3)
by continue host loop on exception

inspired by http://mantis.tokeek.de/view.php?id=637

fix assertion exception in toString of MultiProtocolURL
toString of AnchorURL and MultiProtocolURL are identical code
(no need to override or to protect call to parent)

as reported in https://github.com/yacy/yacy_search_server/issues/43

include initial japaneese translation to language selection

Merge pull request #44 from ImpactCrater/master

Created a translation file ja.lng

Created a translation file ja.lng

I wrote a bit of translation to Japanese.

added peer hash to export dump name.

upd to weupnp-0.1.4.jar

del unused statistic parameter (from status servlet)

enhanced robustnes of image computation

put metadata dc_publisher in corresponding schema field

remove servlet classloder internal cache map (to save the resources, cache hits marginal)
- DefaultServlet includes already a class cache "templateMethodCache" which is emptied 
  on low mem status 
- avoid classloader cache gets has no hits but over time holds all (used) servlet classes 

Also check and index eventual icon url information from metadata.

Display icons in ViewFile "links" mode.

Updated getpageinfo outputs to return page icons list.

format2

format

override detected language (statistic langdetect) only with TLD determided
language if langdetect probability is not high.
+ additionally truncate zh-cn / zh-tw returned by langdetect to 2 char ISO639-1 zh
used by YaCy

include use of condenser's content text for language detection.
Language identification may show poor performance on documents with short or no
title but clear lang indication in text content. Using content text too
improves lang detection.
+ remove double caching of text in Identificator

Correctly handle POSTed parameter also with HTTPS activated

Added favicon url transmission in RWI chunks.

correct isHidden() for smb from last commit 

respect hidden attribute for file and smb directory listing
(hidden directories are not listed, effects crawling of local file system)

Extracted intranet and filtype related rules from getFaviconURL func

compare search page, remove diminished search target
(romso.de, dbpedia.neofonie.de )

upd classpath in batches (remove not necessary htroot)
see prev commit

add findClass() to servlet class loader (used in YaCyDefaltServlet)
In the 2 cases where servlet calls servlet the jvm classloader chain is
invoked and servlet class loaded by jvm loader (successful while requiring 
htroot in system classpath). This patch uses the standard override design
for loaders to handle these cases (making in not longer crucial to have htroot 
in system classpath, as this classLoader is mainly used for servlets and
looks in this case for the class in the configured path).
+ As the default classloader is parallelcapable we should register this too.

Improved plotting

expand initRemoteCrawler() to terminate worker threads if called to deactivate
remote crawl.
On startup we save the resources for remote crawler if disabled. Once started
threads are running idle after disable remote crawl. Now threads are terminated
to save the resources also while disabeling during runtime.
+ remove empty class Channels

support scraping start-/enddate from html tag with property "datetime"
This may be used in html5 <time> tag (which we don't explicite support yet for date in content scraping).

complet resource cleanup of lists in contentscraper's close()

add support parsing swf-metadata to swfparser
flash supports metadata tag in swf file with metadata in xmp (xml) format.
parse some common data to include it in the head section of the html string
of converttohtml.

replace remaining use of java.util.Vector by ArrayList (WebCat-swf)

add support for DEFINEFONT3 (swf8) to webcat parser
experienced issue with JPEGTABLE tag (with length=0) causing abort of parsing (ioexception)
as we don't use/need it for text parsing skip this tag.

add missing swf tag constants according to latest spec
reduce use of synced vector in webcat parser

pass parsed swf content trough htmlscraper
Swf may contain subset of html tags which shoul'd appear as text.
Especially <font> tag may totally screw up metadata servlet if not filtered out.

replace bugfixed webcat-swf.jar

refactor libbuild/GitRevMavenTask (marvenize)
to be able to add additional modules to build

add translation DE,FR submenuRanking.template
upd translation DE RankingSolr_p

add DE translation for submenuCrawler.template
+ upd submenuIndexControl.template

apply remote result count config value to Bookmark Autosearch
+ prepare to make the widely unused Bookmark feature optional 

Merge pull request #40 from Scarfmonster/autocrawl

Automatic crawling

Add missing settings to autocrawl settings page

Add autocrawl settings page

Add the Autocrawl thread

altered clac. of search result items per page to display 
taking the existing limits into account but make it consistent with search option screen for admin and public user
changes:
  - configured default number of items per page (ConfigPortal_p.html) is used as is (no hardcoded limit)
  - otherwise requests are limited to 100 results per page ( = search option, index.html)
      (this basically is the major change, inc. limit from 20 to 100 for public user)
P.S. - the older grant of more (1000), if no online snippet calculation, is kept (for the time being)

see http://mantis.tokeek.de/view.php?id=627 

Corrected access to Load_MediawikiWiki.html and Load_PHPBB3.html

A NullPointerException occured when trying to access theses pages in
Robinson (Search portal) mode

Create autocrawl profiles

remove contendom depending override of search result items per page
initially introduced https://github.com/yacy/yacy_search_server/commit/e4570bffaf6e8c7c71d8945dbed746a91015819d#diff-ae6c130fc11088c830b00ed9256ab56b
(as one part of unexpected difference in actual vs requested results, partial bugfix for http://mantis.tokeek.de/view.php?id=627 )

Merge pull request #1 from yacy/master

Get my fork synced

upd pom javadoc goal
to not fail a build on javadoc errors

fix swfParser reading file signature
before passing to library (current version expects data w/o signature)

update Maven pom - add release-profile
to create the release archive only if profile is activated (speeding up normal compilation)
- bind install of the 2 jar's not available in repository to validate phase (was clean)
  to automatically add these to local repository (with disadvantage it's done every time)

fix tray icon switch
(using predefined/correct config name)

Return more accurate HTTP status 400 with detail message when some error
occurs on ViewImage :
 - missing required parameters
 - url licence invalid

Corrected NullPointerException cases occuring in YJsonResponseWriter
when no description is available.

Updated documented sample url

Improved URLLicence reliability for use by conccurrent non authaurized
users.
Removed URLLicence generation when unnecessary (authorized users)

add de translation for IndexExport_p

fix typo in translation (de,hi) for AccessTracker_p
- rem some not translated in ru (-> currently best maintained translation)

further refactor using standard java / (one) utf-8 charset variable
extending initiative of commit https://github.com/yacy/yacy_search_server/commit/9a2575185027fe822956a4edf538c304862ed0d5 

Merge pull request #28 from Stepanov-Sergey/patch-1

fixed typos

Merge pull request #37 from sudheesh001/LogFix

Log files are commitable and shouldn't be

remove obsolete css style for removed file CacheAdmin_p.html
and remove from translations

add missing quote chars in sk.lng translation file
+ minor: del one redundancy

remove blekko from opensearch config (not available)
see https://blekko.com/
http://searchengineland.com/goodbye-blekko-search-engine-joins-ibms-watson-team-217633

Log files are commitable and shouldn't be

remove workaround for Solr bug regarding multivalued date fields
fixed in 5.4.0
http://issues.apache.org/jira/browse/SOLR-8050

check error URL cache before adding errorDoc to index
- del obsolete related switchboardconstant

remove local credential on download of config files from remote systems
(blacklists, language, skins) to reduce risk to expose md5-pwd.
- remove NoSuchElementException in loop

reintroduce special handling of file upload multipart/form-data from HTTPDemon.parseMultipart
- add filename to parameter fieldname
- add filecontent to special parameter fieldname$file
(some servlets use this $file parameter)

fix for http://mantis.tokeek.de/view.php?id=542

correct filename input in settings_seed_upload.inc form
to get filename
(recently introduced by change form "text" to the more convenient "file" input  https://github.com/yacy/yacy_search_server/commit/50f64ddc3b87c64e6f29dc6f038023f1fb1da7fc)

fix low memory status hint (dht-in disabled)
http://mantis.tokeek.de/view.php?id=619

remove obsolete yacy.init entries interaction.*
(related to removed triplestore)

added greeting line to interactive search and harmonized display
position with 'Administration' line in all administration pages

use curl downloads in download script with silent mode

fix poison mediawikiimporter output queue also after ExecutionException
in worker thread.
Writer of importer keeps needs a poison to close the file. On exception (e.g. OOM)
add a poison marker in outer most try/catch to assure output queue will terminate
in this condition too (and closes+renames the surrogate/in/xxx.prt file)

fix mediawikiimporter number format exception on coordinate parsing
handle uncomplete metadata like "NS=43/50//N". 
For other {expr ... } type entries a try catch added

stub to use a new app launcher for mac

increase http header EXPIRES for css and images in DefaultServlet
to increase browser cache hits for not changing content

skip loading document on crawl start for YMark bookmarks
by adding a constructor giving the already loaded document as parameter.

limit bookmark tag cloud font-size to 2.0em

apply default css styles (class btn) to blacklist* submit buttons 

use input type=file for choosing IndexImportMediawiki_p dump file

fix mime table for ms office / open office documents
(causing wrong parser detect in intranet mode)

change some more submit button to bootstrap btn css class

adjust MultiProtocolURL.protocol detection to handle mailto with "://" in parameters,
and feeding hyperlinks to webgraph processing.

quick fix: go back to display search results favicon via <img> tag and ViewImage,
! until better solution is found !.

Reason: in IE-Browser no individual favicon is displayed with <object> tag 
(always the default) and only few individual fav's with Firefox (randomly)

hint: to be able to use <img src=ViewImage/> return of default icon was
added back to it.

relates to http://mantis.tokeek.de/view.php?id=629

make use of header.getContentType where possible (mime is normalized afterwards)
otherwise use header.mime() differentiated in prev. commit. 

fix HeaderFramework.mime() to strip charset parameter.
Differentiate mime() and getContentType() which gives the raw header field.
This improves parser detection if charsets are included in http content-type field.

fix (todo): correct doc.id of remote search result if no match with newly 
calculated doc hash if different.
Testing showed that in some cases delivered url doesn't match the local
calculated hash. In this case replace doc.id (and host_id_s) with calculation
from url.

exclude in/outboundlinksnofollowcount_i from default schema fields
(not used in any function)

fix: adjust urlstub for mailto links
(skip protocol)

drop returning document language "en" if unknown (fix todo)
which also harmonizes handling of query.modifier for rwi and solr results
(to result must match a given language filter)

show mailto links in ViewFile.html linklist

let html scraper accept html5 <link rel="icon"> for favicon links

Corrected another case of
org.apache.lucene.store.AlreadyClosedException" occuring when
SearchEvent.cleanup() was called while committing local solr index.

prevent mailto links getting into parser result document's in/outbound link collection
by checking mailto scheme early.
- fix upper case mailto protocol assignment
- add test case for getProtocol

Corrected frames preview displaying eventually incorrectly in local
administration mode.

Added an option to eventually limit size of remote solr documents put to
local index. See mantis #626.

fix test htmlParserTest, charset parameter
+ upd maven templating-plugin version

Added a note on deprecated default search field and operator.

Modified MemoryControl.main() test to properly end for better results
displaying.

CachedSolrConnector was replaced by ConcurrentUpdateSolrConnector years
ago - time to let it go
Commented out unused table of cache-objects

fix use of NETWORK_SEARCHVERIFY for rwi verification
was not used to set the searchevent parameter (done in SearchEventCache.getEvent)
- remove unused corresponding QueryParams.filterfailurls param.

remove unused network scanner parameter from queryparameter
Search event is not using networkscanner 
(removed filterscannerfail param always init to false)

add hint to "default max results per page" limit on ConfigPortal
(limit is applied in yacysearch & max. total results by sum result-stack size)
- remove obsolete search.navigation prop (has moved to ConfigSearchPage_p)

Merge pull request #32 from luccioman/master

Fix for MediaWiki import (mantis 625)

Fixed MediaWiki import : DCEntry conversion to SolrInputDocument was
failing. Looks like it was broken since Commit
b43811d38c426ce5237df7d37c88285644e6a544

Updated MediaWimporter main() : display usage in console and stop
properly without calling System.exit

make current ranking score value avail. to search interface / api
Update the result score result field with the result queue ranking value to reflect
the actual calculated/used score,
for rwi & solr stack results.
(calc. etc. is unchanged, it's just that result entry carries the latest val
as api retrieves the number from it)

upd to Bootstrap v3.3.6

Fixed isSolrDump function : PushBackInputStream was not unread when
returning false (for example with a WikiMedia dump).

add missing bootstrap 3.3.4 glyphicons file
see comment @luc http://mantis.tokeek.de/view.php?id=623#c1151

less logging in new language detection

Apply collection query constraint/modifiert to rwi result stack.
Collection is not available in pure rwi entries (but in local solr metadata)
But if user wishes to filter by query constraint also rwi shall adhere to this
(even if only rwi entries with parsed or solr received metadata may fit)
 

Merge pull request #29 from luccioman/master

Modified images render error management

Corrected return type when licence is gone to be consistent with other
error cases.

Corrected error management for unsupported image formats, parsing
errors, and unavailable resources : avoid logging to much Exceptions as
these errors easily occur when searching images.

upd maven pom (add langdetect)

Merge pull request #27 from Stepanov-Sergey/master

added Russian synonyms

urlproxyheader must be in the default package because all classes in the
htroot path must be in the default package

Merge pull request #23 from linkerlin/patch-1

Create .travis.yml

added Russian synonyms

read/init crawl queue in a thread
to speed-up YaCy start on large existing crawler queues

upd to slf4j-1.7.13

remove unused md5 from ViewFile servlet params

fix vsdParser (Visio) parser return statement
(final block un-necessary throw)

remove md5_s from default index fields
it is not assigned a value / not used
Due to above also excluded from transfer protocol.

- No max dimensions specified : render raw image data when source and
target image format are the same.
- Corrected scaling condition.

fix flux factor (additional crawl delay by access count) calculation

throw exception if crawler hostqueue can't create hostpath directory.
In rare cases hostname may not be a valid filesystem directory name,
which can't be created (e.g. containing '*' char). To prevent crawl queue
looping on this invalid entry by throwing a malformedurlexception.

Use same max file size when loading all resource bytes or opening stream
content

Rendering performance improvement : use EncodedImage constructor with
BufferedImage parameter to avoid re-rerendering BufferedImage.

Corrected scaling function for non RGB images.

Refactoring : extracted write InputStream method.

Configuration projet eclipse : ajout nature et validation javascript

Correction erreur de compilation.

start using a template for urlproxy header
It is included as iframe  /proxmsg/urlproxyheader.html
to allow full servlet functionallity and flexibility to display some
index/meta data in future.

Process large or local file images dealing directly with content
InputStream.

If available, check content length before downloading. Check also
content length is not over Integer.MAX_VALUE.

Ensure resource is closed when reading a full file InputStream

have psParser cleanup temp file

set tmpfile.deleteOnExit by default,
to make sure files are removed on shutdown.

Exclude repetitive protocol part in tokenized url 
used as description if none is avail. from parser.

harmonize wordsintitle & CollectionSchema.title_words_val calculation,
remove obsolete partial init of wordreference from urimetadata

add link to quick select blacklist 
from title list

add German translation to re-crawl job

upd to httpcore 4.4.4

fix yacysearch.json "totalResults"
element "totalResults" is included twice (at begin & end), 
only the element after performing the search holds number > 0
see http://mantis.tokeek.de/view.php?id=608

Sort out double keywords (dc_subject) early in parsed documents
- by direct using Set vs. List
- remove not neede String[] getter

Added links to more image test suites.

improve locale translator
- skip empty line
- robustness file section detection (space independant)

do not store subfield *_coordinate + make all num-fields being docvalues

not using 'location' as defaultfacetfield - since we removed it being
default.

simplification / speedup of GenerationMemoryStrategy

do not store subfield *_coordinate

set startuptype of most solr handlers to lazy

fix to not let the AccessTracker-Log grow to much, but have enough data
to monitor.
(+gitignore-correction)

harmonize document title for archive parsers

Create .travis.yml

update bzip and bzip parser process,
to return one document for the file with combined parser results of the
containing file and registers it with supplied url and mime of the archive.

update zip and tar parser process,
to return one document for the file with combined parser results of the
containing files.

optimize order of parsers to try
- start with a parser matching the remote supplied mime

use current tar library for untar files
- remove old source copy

fix tarParser early exit on looping content
- adjust check of data available according to doc 
- return null on no recognized content (to not exit TextParser next parser try)
- use commons.compress directly

fix bzipParser recognition
- Bzip2Inputstream checks magic byte itself to identify bz2 (leave it in input)
- try to suppy fitting mime for parsing bz2 content

increase use of pre.defined CATCHALL_QUERY string

Optimize internal imagequery focus on using content_type to select images
(in favor of url file extension)

Avoid returning an empty image when target encoding is not supported or
when an error occured while encoding.

Updated javadocs for warning on target encoding format potential errors.

Corrected images alpha channel rendering

Made ViewImagePerfTest extend ViewImageTest to ease automated image
render tests

removed totalResults in json to prevent message 'Duplicate key
"totalResults"' when parsing the json in JSONObject (json reference
implementation)

Corrected encoding extension arg parsing

upd readme.mediawiki min java version 1.7

adjust MediaWiki importer geo coordinate calculation
- allow lat/long 0.xxx
- south / west assignment
include test class

fix IndexImportMediawiki_p servlet's refresh header
add url parameter to make sure no parameter are included in refresh url 
which could cause unwanted restart of import job

see http://mantis.tokeek.de/view.php?id=591 comments

fix MediawikiImporter for bz2 dump
skip reading bz2 file magicbyte to identify bz2 format as inputstream reset would be required. Common compress reads and checks the magicbytes internally and throws ioexception if wrong, making preread obsolete.

fix a system.out to log.fine
in bmpParser

remove override of dynamicField coordinate_p in solr schema
(coordinate_p is not a mandatory field as such doesn't need to be declared as schema.field)

fix init of peer flags
(remove hiding of ssl flag)

Created a class to test ViewImage rendering against multiple image
files.

Corrected APNG test suite link name.

Detailed javadoc.

Filled ViewImageTest.html with all remaining IANA image file formats.
Added some links to test suites and specifications.

fix unnececary set null of peer flags, causing reread
remove obsolete version flags

Patch to manage render or load errors is still needed after highlight.js
version upgrade.
Updated patch for better behavior consistency between browsers.

- Keep aspect ratio of images rendered directly by browser such as gif
and svg.
- Corrected quadratic rendering of landscape images with height smaller
than maxHeight

upd javascript img viewerto highslide 4.1.13

Display full size preview using ViewImage Servlet.

Added image preview error management.

Corrected NullPointerException case when ImageIO reader is not found for
image format.

remove obsolete yacy.init entry "secureHttps"
not used anywhere

upd to icu4j-56_1

add a log entry on parsing ajax crawling scheme snapshot 
(prev. commit https://github.com/yacy/yacy_search_server/commit/9252e36aeb3765ba06d4dcf5543ad2e64c70bd4e) 

replaced HashMap with LinkedHashMap to preserve the object order

added log lines

init Recrawl job chunk size to max crawl loader during job start, to use some system preferences
and allow injection of recrawl urls before queue is empty
During recrawl the balancer hangs on the very last urls often on hosts with huge delay time,
by allowing injection earlier progress is more balanced. Max number of injected crawl urls by recrawl job is 2 * max loader.

Created a generic ViewImage performance render test.

Created a ViewImage rendering performance measurement test.

Refactoring : split into sub-functions to make it understanding and
performance measurement easier.

Updated table headers and SVG file url for case sensitive OS.

unescape MultiProtocolURL getAttributes() return values.
use getAttributes() to get query parameters as clear text (w/o url encoding)
use getSearchpartMap() to get in internal format (url encoded)

fix for http://mantis.tokeek.de/view.php?id=606

refactor special handling (static override) of SUPPORTED_EXTENSIONS/MIME_TYPES
not used for genericImageParser

add links with image extension not automatically to image links.
With the wide spread use e.g. of Wikimedia the url file extension of links with image extension often point to html.

Added ico and bmp sample pictures

Added JPEG 2000 and FITS samples

Added image formats and informations for each format.

handle image preview for url w empty file extension
fix of commit 688f7b2a5c194124dc2a253be45a1af1b814bef2

check jpeg file signature in genericImageParser
to fail early without further object allocation if source is not a jpeg.

use recrawljob w/o sort results by date
This is a workaround for existing index (not fully reindexed) since intro of schema with docvalues
to prevent solr exception causing recrawljob to fail with
org.apache.solr.core.SolrCore java.lang.IllegalStateException: unexpected docvalues type NONE for field 'load_date_dt' (expected=NUMERIC). Use UninvertingReader or index with docvalues.

removed unused import

upd to poi-3.13

Created a html test page to check ViewImage rendering with different
file formats.

allow/display svg images in image results previews
svg is not supported by awt but by most browser. Image content is delivered as received (without size adjustment)

remove some unused var allocation in parser

follow-up to latest commit: also flush the search cache if all crawls
had been terminated.

every time a crawl is started, the user expects a different search
result behaviour. This requires that the search cache is flushed for
each crawl start. TODO: this should also be done if a crawl is
terminated.

in case that the include_string contains several entries including
1-char tokens and also more-than-1-char tokens, then remove the 1-char
tokens to prevent that we are to strict. This will make it possible to
be a bit more fuzzy in the search where it is appropriate.

add also 1-character tokens to the token list because that could be also
searched for. A full-string search for a filename may fail if those
1-char tokens are omitted

add a end condition to svgParser for wrong content
(if parser choosen just by file extension)

remove double caching of inputstream in ViewImage

fix old/obsolete solr dependency to stax
delete obsolete jar

Add report profile with OWASP Dependency-Check to maven pom

remove rdfParser from init (current function identical with genericParser)

add svgParser to parse metadate from svg images
Reads document level included title and description and skips the graphic content to save bandwidth.
svg metadata element is not interpreted
- remove rdfParser from init (current function identical with genericParser)

optimize parseInt for <img> tag attribute parsing
Performance better as using Numberformat.parse or parseInt(substring())

check for loading error (includs unsupported formats)
to prevent blank thumbnail display in image search because of not handled source which don't load on click.
Now the cross icon indicates the problem (inlcuding not supported format)

Correction for mantis 535: inurl: parameter doesn't work on URLs with
upper-case letters

harmonize/correct assignment to Ymarkmeta.mime
replace use of deprecated

Fix for index entries which have id's not computed as hash from the url.
This makes it possible to operate with outside-computed url hashes in
enterprise environments not using the build-in crawler from YaCy.

remove unused check for known fileextension in searchtrailer
(check is done on add to filetype-nav)

optionally include mime in p2p url exchange string
if doctype decodes to ambiguous mime and default conversion is not equal to original
 

add Portuguese month names to date recognition

fix html parser taking <style> content as text.
Noticed some result description contain css content from style tag.
Added <style> to tag list to scrape it's content not as text
+ test case included

patch for a bug inside of solr since solr 5.0 when using a boost
function with a numeric date field:
"unexpected docvalues type NUMERIC for field 'last_modified' (expected
one of [SORTED, SORTED_SET]). Use UninvertingReader or index with
docvalues."
This is a well-known bug inside solr which prevents that now the 'sort
by date' in the YaCy search interface can be used. Without this patch no
results at all is displayed (since the exception prevents that). Now
there is at least a result but it is not ordered properly.

limit css scrolling to result/content window x
from pull request #10

Merge pull request #10 from Raegdan/raegdan-css-layout-fix

Fixed CSS scrolling

Hack to prevent Solr issue on partial update on a document containing multivalued date field
(regardless if these fields part of update).
Switch partial update option off in postprocessing if schema contains *_dts (multivalued date field).
see http://mantis.tokeek.de/view.php?id=601

adapt SolrServerConnector.add to handle error on partial update input document.
In case of error we deleted the original document and added the new doc to the index.
This is not valid for partial update documents (which contain only a subset of the fields).
Remove the "delete" error handling step.

add test case for partial update - to discover effect on YaCy for update of documents with multivalued date fields (like dates_in_content_dts) 
current result: loss of fields/information in index document, see EmbeddedSolrConnectorTest.testUdate_withMultivaluedDateField()

on reindex delete index document with invalid url
if discovered

use a parsed date in Document.toString

Returned again to main repository location : does anyone want to
consider mantis 597 ?  (http://mantis.tokeek.de/view.php?id=597)

Merge from main repository

improve filtering by filetype navigator.
The used url-filter for filetype doesn't require ".ext" resulting in too many matches,
add a sort-out filter for RWI results.

prevent metadata records in index w/o valid url
by throwing MalformedURL exception on URIMetadataNode creation

extract modification date from vCard (vcfParser)

extract lastmodified from openoffice doc
set lastmod date in office document parsers

in case that a site crawl is started for urls with file:// path, the
host filter does not work because there is no host given in such urls.
In that case, patch the filter to be a sub-path filter.

fix exception throw after sendError in DefaultServlet
- reduce debug exception logs in crawler

Merge pull request #12 from luccioman/master

Updated french locale and added new translator utils

Return to mai repository version

Added utils to help translation without launching full YaCy application
:
- translate all source files with a locale
- list all non translated files with a locale

Added a function to list files recursively.

Translator refactoring : 
- deleted useless new StringBuilder allocation
- use of a new reusable FileNameFilter
- added javadoc

fix missing license in image search
see http://mantis.tokeek.de/view.php?id=522

Updated french translations for yacysearhitem.html,
yacysearchtrailer.html and Steering.html files.
Corrected various labels.

Corrected br markup

Corrected bookmark link title

fix missing onclick in ConfigPortal
to enable checkbox

ignore /DATA (Eclipse Mars)

apply same size constrain on result image from doc
as for linked images
see https://github.com/yacy/yacy_search_server/commit/19f1308bf09172d2be66c58289d52ba2b2c0cf9d

enable Solr schema dynamicField _p (type=location) for YaCy coordinate_p field

complete TODO: getFileExtension handle dot in query part
+ testcase

French Translation update by Luc:
http://forum.yacy-websuche.de/viewtopic.php?f=8&t=5671

start recording/indexing pixel size for image document
as for linked images

check mime prior to ext for metadata modification for images

enforce th result images limit to > 16x16px
for linked images
http://mantis.tokeek.de/view.php?id=594

Updated french translation for index.html, yacysearch.html and
simpleheader.template. Correcte special characters to use HTML entities
instead.

remove exired domain titan.deep-one.in from bootstrap.seedlist

Updated translation of index.html, yacysearch.html and
simpleheader.template, corrected some special characters not written as
HTML entities.

fix NPE on .yacyh result url of disconnected peer
(cleanup yacyshare remaining)

log missing seed.port
in favour of exception to prevent repeating throws

fix: Preserve protocol in url proxy 
to connect to http/https. Display warning if https target is viewed over http

upd to jsoup-1.8.3

added / corrected charste to be 1.7 compatible.
@Orbiter: please check is this is ok for you

exclude more default search fields from  text copy to text_t
for metadata index documents

remove obsolete interface SearchAccumulator
and unused SRURSSConnector Thread inheritance

enhanced logging

! finish running crawls before applying !
Allow crawl urls up to 2048 character 
fix for http://mantis.tokeek.de/view.php?id=575

use some more declared HeaderFramework constants

add missing ; in base.css

pom: have Maven dependency management decide on transitive Lucene dependencies

replace deprecated myPublicLocalIP() in AbstractRemoteHandler

remove unused Transmission hit counter

use more absolute path for config file opening
as suggested in pull request 5 (https://github.com/yacy/yacy_search_server/pull/5)

added bayes filter from Philipp Nolte, originally taken from
https://github.com/ptnplanet/Java-Naive-Bayes-Classifier
and modified inside the loklak.org project. After optimization in loklak
it was inserted into the net.yacy.cora.bayes package. It shall be used
to create custom search navigation filters.

The original copyright notice was copied from the README.md from
https://github.com/ptnplanet/Java-Naive-Bayes-Classifier/blob/master/README.md
The original package domain was
de.daslaboratorium.machinelearning.classifier

using latest enhanced (un/)gzip methods from loklak for yacy

added export option to export the fulltext of the search index text only

try a healing of the cache if the index file is corrupted

added log lines for query performance profiling

upd to SLF4J-1.7.12

upd to httpclient-4.5 and httpmime-4.5

upd to icu4j-55.1.jar

upd to jsch-0.1.53.jar

Fixed CSS scrolling

When the sidebar on search page becomes scrollable, the scrollbar shrinks the sidebar and makes the search results weirdly scrollable on X axis by several pixels. Now the sidebar always have a scrollbar, and results are never X-scrollable.

upd to lib/weupnp-0.1.3.jar

added jsonp to suggest servlet

upd NB classpath

remove old vocabularies and synonyms before adding new

added another default network / commented out

added msg (text emails) format; should be handled by html parser.

fix one implicit Integer/Long type conversion
-> causes Java 1.8 compile error

add CommonPattern for multiple spaces 
to eliminate empty split words on following spaces

added enrichment of synonyms and vocabularies for imported documents
during surrogate reading: those attributes from the dump are removed
during the import process and replaced by new detected attributes
according to the setting of the YaCy peer.
This may cause that all such attributes are removed if the importing
peer has no synonyms and/or no vocabularies defined.

refactoring: separated condenser and tokenizer

Rem depreciated AdminHandlers in solrconfig.xml
avoid warning log
W  org.apache.solr.handler.admin.AdminHandlers <requestHandler name="/admin/"  class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore

fix for non-authorized view of IndexBrowser: show only the number of
non-failure documents

enhanced surrogate import process speed (dramatically!)

fix for
- bad regex computation for crawl start from file (limitation on domain
did not work)
- servlet error when starting crawl from a large list of urls

suppress access to solr when doing search suggestions in case that the
index has more than two million documents. This protects the index from
beeing flooded with search requests that cannot be resolved before the
real search query has to be computet.

remove redundant code

Next Try for a fix for upload-connection staying in blocked state.
This was caused by reading via GZIP from close-wait connection an caused
high cpu- and system-loads.
Instat of implementing handling of the RedListener now I found a
timelimeted 'get' "realy" solving this problem.

Resourceobserver log warning - deleting releases files - only on actual deletes
instead of entering routine

Fix for upload-connection staying in blocked state.
This was caused by reading via GZIP from close-wait connection an caused
high cpu- and system-loads.
Solved by implementing handling of the RedListener.

add log entry on release file delete by ResourceObserver

implement deleteOldDownloads in RexourceObserver on low diskspace
- direct assign sb.observer (skip redundant InitThread)

added link to Snapshots in search results if the snapshot exists and
option is set in ConfigSearchPage_p
(this is a stub: we also need a visualization of pdf files!)

enhance recrawl job
- allow to modify the query to select documents to  process (after job has started)
- allow to include failed urls (httpstatus <> 200)

servlet for latest commit

upd to poi-3.12.jar

remove augmented parsing activation from frontend
experimental implementation not used and based on error prone experimental rdfaparser 

remove RDFa parser activation from frontend
reason: experimental implementatin of RDFa parser not executed (limited to special urls) but may cause error on normal html parsing due to a inputstream.reset

removed the new index export method from the IndexControlURLs_p.html
servlet and moved it to a new /IndexExport_p.html servlet. This servlet
is now more prominent linked in the main menu under Production -> Index
Export/Import

Merge origin/master

remove obsolete searchfl work table
was used to register urls with not complete words in snippet but is never accessed

correct the dark themes to show also a dark navbar on searchresults

gzip compression will perform more efficient and with better compression
level

full solr xml exports will now be automatically compressed during
export. That makes it possible to export a solr xml dump even if disc
space is low.

wrap HeaReader close() in a catch Throwable block to prevent that an
excpetion during close blocks the whole shotdown process

added surrogate import process for exported solr dumps.
Just throw your solr dump file into DATA/SURROGATES/in/ and it will be
imported!

prevent disc usage when showing tray animation

re-licensing some of my old visualization classes under LGPL 2.1

adding a 3-pixel font generator made some time ago..

All entities of crawl profiles are now editable in the crawl profile
editor.

- Image search expand box, adjust javascript hs padtominsize parameter, to make sure expand box doesn't shrink on small images
- asure ImageResult.imagetext has value for the link text (use filename if no alt text given)

remove hard throw exception in makeResultEntry
remove not used "share." peername.yacy url rewrite

use available mime (instead null) on imageresult from metadatanode

revert deletion of BinSearch
(accident)

Eleminate duplication of values for search ResultEntry 
by instatiation from URIMetadataNode, by eleminating differentiation of ResultEntry/URIMetadataNode.
- moved remaining ResultEntry functionallity to URIMetadataNode
   - for 1:1 functionallity added a function makeResultEntry() 
- removed ResultEntry 
- refactored related code

Main difference is after makeResultEntry the text_t content is removed and alternative title/url strings for display are calculated.


Main difference left is, that 

fix compiler notification of missing serialID 
from last commit

refactor ResultEntry to be based on MetadataNode/SolrDocument
to share/reuse common access routines

Implement sharing of ioDispatcher for term & citation index
as proposed in ioDispatcher description

use doctype() in ViewFile to choose display routines
in preference of getfileExtension()

On imageSearch prefere mime to sort out none-image documents
Generalize the hack to prevent urls with just a img extension beeing returned

improving http://mantis.tokeek.de/view.php?id=528

improve MultiprotocolURL.getFileExtension()
prevent string OOB while querypart contains a dot (return just "")
see log snippet in http://mantis.tokeek.de/view.php?id=533

Increase IODspatcher dumpQueue size to 2 to reduce risk of concurrent emergency dump,
skip concurrent emergency merge
dealing with/see  http://mantis.tokeek.de/view.php?id=566

fix string OoB on getImagelinks with long alttext
in description calculation

Convert content charset for display via CacheResource_p
Cached resource charset encoding might not fit to internal handling (using utf-8),
convert resource to utf-8
see http://mantis.tokeek.de/view.php?id=576

removed a -UNRESOLVED_PATTERN-

Limit extra sleep of BusyThread on LowMemCycle

detail optimization of RecrawlThread

Initial (experimental) implementation of index update/re-crawl job
added to IndexReIndexMonitor_p.html
Selects existing documents from index and feeds it to the crawler.
currently only the field fresh_date_dt is used determine documents for recrawl (fresh_date_dt:[* TO NOW-1DAY]
Documents are  added in small chunks (200) to the crawler, only if no other crawl is running.

correct log msg text

fix extract of inboundlinks_protocol_sxt
url counter maybe > 999

fix early return in addToCrawler
check / handle all supplied urls after error url

fix for latest commit, see
https://github.com/yacy/yacy_search_server/commit/f810915717579d490259d70610dc4118b7c6e6e9#commitcomment-11145880

fix NPE on MultiProtocolURL on url with parameter value and '='
in getAttribute
- added test case for it

added crawl start from a clone with very, very large url: they are now
encoded as post submit form inside a javascript creation function.

enable api calls with very long urls

upd library reference of missing jsch-0.1.21 in seeduploadscp.xml
upd to jsch-0.1.52.jar

add opensearch rss results to dht collection (due to text = snippet)
which is used to differentiate meta from full data
- make sure check for dht is not dependant on number of collection entries

add bookmark.query to edit form

persist bookmark timestamp
on setTimeStamp()

upd to commons-io-2.4.jar

update bookmark autosearch description
- add german translation

added option to re-index exported xml snapshot dumps to
HTCACHE/snapshots by just placing them in the SURROGATES/in path

revert of 8a7c68e4c7f6a682e3ef656b423ce1ad76b42caa
keeping surrogates after processing is essential for some users. If the
space they are taking is too high, please set up an automatic deletion
process (like a cronjob).

added must-not-match filter to snapshot generation.
also: fixed some bugs

adding a try-catch to link graph processing to prevent that a single
malformed url interrupts the storage process

on bookmaring of search result, remember orig. query in separate bookmark property
(instead of using the description field)
- adjust display and autosearch
- don't overwrite existing bookmark but combine info

break out of NormalizeDistributor loop on timeout

harmonize filesearch input box layout

upd to metadata-extractor-2.8.1

upd to poi-3.11.jar

fix typo in image filter query
(extra bracket)

fix String out of range in Collection Nav
see http://mantis.tokeek.de/view.php?id=573

improve character encoding for urlproxy servlet
for none utf-8 pages

upd to jsoup-1.8.2

make Quality of Service Servlet available to prioritize requests from local host
This assigns priorities to incoming requests. Higher priority numbers are served before lower.
(disabled by default in defaults/web.xml, 
uncomment or copy entry to DATA/Settings/web.xml)

correct typo in de.lng

upd parser calls in test cases

add additional links to crawl queue pages

don't record dump generation calls since that
- is not a change of the index
- happens very often within self-backup strategies from the outside
(i.e. cronjobs)

Merge pull request #4 from dertuxmalwieder/master

Readme improvements

Readme improvements

Now GitHub should display it properly. Also, added OpenBSD.

Merge pull request #2 from Scarfmonster/master

English Synonyms and small fixes

Merge pull request #3 from shaman/master

How about normal font weight in the searched titles & fix RSS icon position?

fix typos

add vertical margin to the search cloud block

fix RSS icon displaying

increase view space with normal font weight in the searched titles

fix NPE in Vocabulary_p servlet
called w/o parameter

fix for Accept '?' URLs column in Crawl Profile List

SynonymLibrary status check fix for multiple files

added English synonyms

skip redundant add. of keywords to text
search uses keywords as default search field

put https port in peers dna
as we flag if a peer is accesible via https, we need to know the port if we want to use is (e.g. for interYaCy communication)
start to provide / tansport the port by recording it in peers dna.
- add https link on the Network.html lock symbol

incoming connection count/text fix
improvement on http://mantis.tokeek.de/view.php?id=570

Merge pull request #1 from Scarfmonster/master

Search navigation fix

add info text icon next to Augmented Browsing check-box
with hint to config page

show "Augmented Browsing" link in search result only if urlproxy allowed and option switched on in layout
(AugmentedBrowsing_p.html, ConfigSearchPage_p.html)
as user only gets a error page if the option is not enabled

Search navigation fix

added parsing of contentprop attribute in html tags for
content='startDate' and content='endDate'. The value of these field is
now written to new solr fields startDates_dts and endDates_dts.

reverted json syntax for facet results to version from january

added parsing of dd, dt and article html fields. The parsed result is
written to special solr fields which are deactivated by default.

more logging during start-up

<experimental> added parsing of <article> html element.
Whenever such an element occurs, the complete content of all article
elements replaces the parsed <content> part of documents.

enhanced suggestions

replaced "fork me" banner with github banner

upd to httpcore 4.4.1

split query into filter query and text query to get better ranking
results and faster results

testing switching off cold searchers; maybe this brings performance
enhancements when using large facets

when selecting collections in navigation, do show the un-selected
collections in search result. When selecting one of them in another
search, switch off the previously selected collection. This actually
turns the collection navigation modifier into a radio-button like
behaviour

replace deprecated getIP with getIPs in AbstractRemoteHandler

include htroot (*.class) in maven clean 
harmonize antrun javac call in pom with build.xml

add SynonymLibrary status to DictionaryLoader_p servlet 
http://mantis.tokeek.de/view.php?id=564

refactoring of filter queries (several queries instead only one)

show location nav as selectable nav in search page layout
- switch automatically on upon load of geodata provider
- but allow switch on also without geodata file (and display the location nav if search result has lat/lon location)

use a cursor hand on facet headline to show that this is clickable

added an expansion option to search facets on result page:
- if less or equal of 8 facet options are present, they are shown by
default
- if more facet options are present, they are hidden
To view or hide all facets, just click on the facet header bar

make location facet return results
for location nav facet of field coordinate_p does not return results, now using coordinate_p_0_coordinate as alternative to get facet counts. As the actual facet value is not used this should not harm any analysis (even if facet is a incomplete location).
If facet value is used in future likely *_geohash field could be introduced (for facet and other ... as transport value)

fix display and limit of max server connections after startup
(on restart value returned to default=50)
This has no effect on Jetty but the limit is still respected.

add err msg on failure during Load_rss

correct percent encoding for '%' char

added a new collection type 'dht' to all documents from the peer-to-peer
interface to distinguish rich and poor document data.
This also reverts some changes from commit
796770e070daf38289b594f4cbdc65b9ce0ca2b1 because the firstSeen database
is the wrong method to distinguish these types of data

fix missing display of   CrawlerMonitor -> robots.txt Monitor
revert delete of file api/table_p.html see https://gitorious.org/yacy/rc1/commit/3ffe19b85cb6f2c1d26922eb61ca73a42096e8e9
(still used in this menu)

Updated Git links from Gitorious to Github.

prevent overwrite of crawled or received full documents by (newer) metadata
To protect rich index data (full resource) from overwriting by metadata gathered during remote search,
the newly introduced "firstSeen" index is used to differentiate between full-resource-doc and metadata,
as a "firstSeen" entry is only added on store's of full-resource-docs (during crawl or remote search).

Fixes hanging FlushThread (see
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5447)
by replacing put() method by the more robust add() to
add a merge job to the queue.

fix snippet containig number with comma as desmo point  http://mantis.tokeek.de/view.php?id=344
to keep it as one word (by altering the split regex)
- added sniipet test case with number
- regex for word split to match multiple splitcars

fix error on *abc query input
http://mantis.tokeek.de/view.php?id=486

apply UTF-8 encoding
copied from escape()

fix for path with char code > 255 
(causing index out of bound exception)
+ test cas for it

fix url encoding for path see http://mantis.tokeek.de/view.php?id=559
So far we used same escape procedure for all parts of the url (which includes x-www-form-urlencoded for all url components)
Added capability to use different encoding rules for the different url components (through specific bitset for each component).
(this is inspired by org.apache.http.client and java.net.uri implementation).
- Added test case for  http://mantis.tokeek.de/view.php?id=559

try to fix hang on index blob merge on shutdown
http://mantis.tokeek.de/view.php?id=505
It happens but not able to reproduce. This change makes sure terminate signal is catched at end of currently running merge jobs

fix url (path) %-decoding http://mantis.tokeek.de/view.php?id=519
- add test case for this

update configheuristics_p.html text 
to state current opensearch heuristic function

fix version conflict in pom
for commons-io

exclude default search fields from  text copy to text_t
for metadata index documents (reduce text redundance)

For remote crawlReceipt add document abstract/description
enhance the returned metadata returned to the originator by description_txt to improve fulltext search result hits.

harmonize snippet computation
to considere description_txt always (solr hl & internal).
For now just added desc to text list for computation, could be further equalized with hl computation.

remove unused statement

added special terms for on: - Date modifier: tomorrow, today; i.e.:
search for: "Berlin on:tomorrow" to find events happening tomorrow in
Berlin

generalized time period computations

add test case for snippet html encoding check

upd pom

postpone raw snippet html encoding upon use
instead of during init of snippet 
adressing http://mantis.tokeek.de/view.php?id=551

apply query parameter getQueryFields() to GSA servlet

Next try to fix start script for OpenBSD.

fix mimetype of rss items in rss parser
- remove self reference as anchor for items

enhanced date parsing time

introduce getQueryFields to return default query fields (queryparamter QF)
calculated from boostfields config, making sure title, description, keywords and content is always searched.
- apply change to solrServlet makes sure every remote query uses at least all locally defined boost fields for search
- apply to local solr search
- simplify select query by using QF defaults

add description_txt to default query fields,
Dublin Core Metadata field extracted by most parsers.

add extracted description/subject to pptParser

url unescape add check for inconsistent utf8 multibyte parsing
If the url contains special chars (like umlaute äöü) it's interpreted as multybyte char and actually not converted at all (removed).
Added a check if the multibyte convesion is not complete, just add the char as is.

This fixes http://mantis.tokeek.de/view.php?id=200

stop sending crawl receipts if receiver got offline

upd lucene api doc link

add extracted description/subject to docParser 

replace depreciated HTTPClient setStaleConnectionCheckEnabled with setValidateAfterInactivity()

replace depriciated HTTPClient ALLOW_ALL_HOSTNAME_VERIFIER with NoopHostnameVerifier()

fix formatting issue if snippet contains html code
replacement for reverted commit
https://gitorious.org/yacy/rc1/commit/61f42a792872c7fc83c7d4d64e1f0bb9d063c810

upd to commons-codec-1.10.jar, commons-compress-1.9.jar

revert: formatting fix eats also up highlighting
need other solution for snippets with unwanted html code 

upd to httpclient-4.4

fix formatting issue in search result display 
if description contains html code
noticed e.g. for id=NmNdJ9uApLaQ  http://hswong3i.net/blog/hswong3i/virtualmin-drupal-7-x-ubuntu-12-04-howto

allow/recognize host in file: protocol crawl target
This is useful in intranet indexing while crawling a intranet file server accessed via hostname while e.g. under Windows mapped to different drive letters on individual clients.
Here you can crawl e.g.  file://fileserver/documents having a valid uri in that intranet environment (while e.g. P:/documents might be client dependant).

fix parser test cases
(Vocabulary paramete)

disable optimistic GC assumption in StandardMemoryStrategy
After several tests found that eom is not prevented. Major reason in testing was assumption future GC will free avg of last 5 GC.
Disabeling this check improved eom exceptions.

Added simplest testcase used for verification

the cleanup process experienced a 100% CPU load situation and the loop
did not terminate:

Occurrences: 100
at java.util.HashMap$KeyIterator.next(HashMap.java:956)
at
net.yacy.cora.protocol.ConnectionInfo.cleanup(ConnectionInfo.java:300)
at
net.yacy.cora.protocol.ConnectionInfo.cleanUp(ConnectionInfo.java:293)
at net.yacy.search.Switchboard.cleanupJob(Switchboard.java:2212)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:105)
at
net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:215)

This tries to fix the problem; the problem should be monitored

hack to make date detection faster (while it becomes a bit incomplete
regarding language alternatives)

enhanced suggest function

fix Umlaut handling in blekko heuristic search term
http://mantis.tokeek.de/view.php?id=169
observation: blekko seams to block xxxbot agents (=0 results)

url with semicolon or comma handling in proxy request
apply patch supplied with bugreport http://mantis.tokeek.de/view.php?id=540

small correction for last commit

upd error message for proxy
fix http://mantis.tokeek.de/view.php?id=539

remove remote indexing option in crawl start if not in p2p mode

adjust table column width to not line wrap crawler traffic line

cloning a crawl now accepts the class name of vocabulary scapers

configuration option for maxload limit for remote search

add shortMemory check to heuristic search 
and skip operation on shortMemory (no request to remote openserch systems)

fix: malformed filename in image search
fix for http://mantis.tokeek.de/view.php?id=533

refactor: just some more useages of constant for term ":[* TO *]"

remove hardcoded initialization of language nav if not used

Added & in start script for *NIX which was lost a few commits ago.

refactoring of autotagging code (combined same code pieces)

enhanced initialization speed of vocabularies by using better
normalization and by removal of unused data structures

using precompiled CommonPattern.TAB for split

using precompiled pattern CommonPattern.SEMICOLON for splits

persistency for vocabulary facet switch

introducting a new getConfig method which parses comma-separated llists
from setting fields; refactoring for all places where such lists are
parsed

refactoring with CommonPattern.COMMA

do not reindex based on vocabulary fields (there are meanwhile many of
them) and some default settings

refactoring of reindexSolr (just replaced constant string)

Allow to hide linkstructure graphic in crawl monitor 
using/setting the config param DECORATION_GRAFICS_LINKSTRUCTURE

removed some warnings

the LinkedBlockingQueue is much faster than the ArrayBlockingQueue
(strange but this is the result of a test:
ArrayBlockingQueue: 39461 lines / second;
LinkedBlockingQueue: 60774 lines / second)

fix: eom on parsing ico file by genericImageParser

trace: java.lang.OutOfMemoryError: Java heap space
	at java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
	at java.awt.image.Raster.createPackedRaster(Raster.java:467)
	at java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
	at java.awt.image.BufferedImage.<init>(BufferedImage.java:331)
	at net.yacy.document.parser.images.bmpParser$IMAGEMAP.<init>(bmpParser.java:149)
	at net.yacy.document.parser.images.bmpParser.parse(bmpParser.java:69)
	at net.yacy.document.parser.images.genericImageParser.parse(genericImageParser.java:116)

update to latest code changes from json.org

Let auto-disabled crawls recover if low resource condition vanished.
Analog to autodisabled DHT switch autodisabled crawls back on upon mem ok
by remembering the autodisable by conf parameter.

write java version to status page

new development cycle