YaCy Release 1.7

- added d3.js for visualizations using embedded svg
- added a servlet api/linkstructure.json which generates a link graph
information in json
- added a javascript link graph renderer hypertree.js using d3 and the
new servlet linkstructure.json
- embedded the new link graph in the crawler monitor and the host
browser

Complete redesign of administration top-level menu. This follows two
principles:
- provide an easy tutorial-like "what should I do first" menu
- provide all elements which are subject to most first questions to YaCy
exibition people on top level: Resource limitation, Parser and Ranking
settings
I apologize to everyone who are used to the old style and need to find
the menu items (again) after this change. I hope that this will make the
interface more usable for new users who see a web indexer/crawler the
first time.

redesign of search page integration menu structure

replacing direct html table cellspacing with css set-up for cellspacing

alternative UrlProxyServlet (for /proxy.html) using different url rewrite rules 
- use JSoup parser for selective rewrite of html body <a href=  links only,
instead of regex which rewrites also header href/src links
- this improves display of pages which use header <base> tag
- tags with src attribute are taken from original location (like css) improving display and are not routed trough the indexer
Disadvantage: scripting links will drop out of proxy

Setting of the servlet through web.xml exclusivly (in case one would like to quickly switch back to the YaCyProxyServlet,
leaving the existing code of YaCyProxyServlet untouched available)

added missing @Override annotation

- added new input field in index.html
- changed progress bar in yacysearch.html
- moved pagination navigation to page bottom
- moved search term input field to headline

replaced pdblue-homebrew buttons with bootstrap standard buttons

removed underline for anchor refs and replaced it with
underline-on-hover. Please update your style (switch and switch back)

- removed usage of jquery and yacy/ui in standard search interfaces as
possible
- re-implemented formerly jquery-based tagcloud with simple html
- fixed minimized menu for mobile/minimized view (try that: make window
very narrow...)

- added bootstrap css framework
- adopted all YaCy administration pages to new framework
- created new search page layout (working, but still work in progress)
- old skin files are fully appliable! (and looking good)
- target is a new style based on bootstrap examples, see /test.html
- icons in YaCy may be replaced by glyphicons (to be done)

- added concurrency to postprocessing of webgraph document
- bundeled separate webgraph postprocesing steps into one

enhanced concurrent loading by using a fixed set of concurrent loader
processes in favor of throwaway-processes. The control mechanism does
less often report a 'queue full' message to the busy loop which then
does not perform a long busy waiting; instead all requests are queued
and new loader processes are started if necessary up to a given limit
(as set before)

attaching names to all Threads to get a better view in profiling tools
like VisualVM

- enhanced postprocessing speed and memory footprint (by using HashMaps
instead of TreeMaps)
- enhanced memory footprint of database indexes (by introduction of
optimize calls)
- optimize calls shrink the amount of used memory for index sets if they
are not changed afterwards any more

bump to httpclient-4.3.3

do not use luke requests for remote solr servers if the result is
different from normal requests. This happens if the remote solr is
actually a solrCloud; in such cases the luke request returns only the
result of the single solr peer, not the whole cloud.
also done: some refactoring.

Merge origin/master

changed strategy to test existence of documents in Solr: using the
update time. The reason for that is a better caching for the crawler
double-check, which needs the update time for crawler steering.

enhanced the resource observer
The resource observer is now able to recognize free disk space AND
available space for YaCy. The amount of space which is assigned for YaCy
are defined in new settings in the configuration file.
Furthermore, there is now a cleanup process which deletes files in case
that an autodelete is activated. The autodelete is now BY DEFAULT ON if
the disk space is low, which means that YaCy starts to delete documents
when the disk is full!

clear solr caches in case that an exception occurrs. The reason behind
this hack is the occurrence of Exceptions like:
W 2014/02/11 18:51:33 ConcurrentLog GC overhead limit exceeded
java.io.IOException: GC overhead limit exceeded
        at
net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:334)
        at
net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getDocumentById(MirrorSolrConnector.java:173)
        at
net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getDocumentById(ConcurrentUpdateSolrConnector.java:415)
        at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:331)
        at net.yacy.search.index.Fulltext.getMetadata(Fulltext.java:317)
        at
net.yacy.search.query.SearchEvent.pullOneRWI(SearchEvent.java:1024)
        at
net.yacy.search.query.SearchEvent.pullOneFilteredFromRWI(SearchEvent.java:1047)
        at
net.yacy.search.query.SearchEvent$3.run(SearchEvent.java:1263)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:3077)
        at java.lang.StringCoding.decode(StringCoding.java:196)
        at java.lang.String.<init>(String.java:491)
        at java.lang.String.<init>(String.java:547)
        at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:187)
        at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:351)
        at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:276)
        at
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110)
        at
org.apache.lucene.index.IndexReader.document(IndexReader.java:436)
        at
org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:657)
        at
net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.SolrQueryResponse2SolrDocumentList(EmbeddedSolrConnector.java:230)
        at
net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getDocumentListByParams(EmbeddedSolrConnector.java:320)
        at
net.yacy.cora.federate.solr.connector.AbstractSolrConnector.getDocumentById(AbstractSolrConnector.java:330)
        ... 7 more
        
This problem was analysed with the Eclipse Memory Analyser after a heap
dump, where the following problem was reported as the main Problem
Suspect:

One instance of "org.apache.solr.util.ConcurrentLRUCache" loaded by
"sun.misc.Launcher$AppClassLoader @ 0x42e940a0" occupies 902.898.256
(61,80%) bytes. The memory is accumulated in one instance of
"java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system
class loader>".

This memory is part of the result cache of Solr. Flushing this cache
appears the most appropriate solution to that problem.

make internal page links relative to ease any future development for context aware servlets
note also http://bugs.yacy.net/view.php?id=106

replaced solr 4.6.0. with 4.6.1

fixed a bug which prevented crawl starts after a network switch

fix for linkstructure computation: now also detecting dead links

another fix for inbound/outbound detection

NPE fix

fix for parsing of numeric value in case that boolean values are given

fixed thread dump in case of wrong seeds

bugfix to inbound/outbound identification

fix for first start options and added german translation for popup texts

fixed crawl start path for cloned crawls

fix for naming bug

npe fix

fix for crawler column width

fix for paths in locale files

fixed crawl start links after renaming them in latest commit

fix: 3 more links pointing to old /xml path

fix: blacklist xml export path to xml template

fixed visibility of custom icon

fix for relative sitemap urls

added close to fix possible resource leak warning

fix: button style IndexReIndexMonitor_p.html

fix: proxyservlet path header setting

one more Thread Dump fix for new bootstrap css style

fix for thread dump line spacing

fixed auto-activation of search field

fix for source and target clickdepth in webgraph index

fixed search field

misc style bugfixes

fix for page navigation bar

fixes to progess bar

fixed bold label setting in bootstrap which causes a bad look in YaCy

fixes to locale after renaming of some entities in online interface

fix: set YaCy CONNECTION_PROP_HOST  Header in ProxyServlet to host incl. port

prevent exception on Site Crawl if no start url is given

fix for bad/missing values of size_i

fixed another shutdown problem (only occurs if webgraph core is enabled)

bugfix-bugfix

bugfix to proxy handler: recognize the own yacyh-host

NPE fix for Thread name (just commited yesterday, sorry)

fixes and enhancements for postprocessing

another fix for the shutdown exceptions

fix: de translation of menu link

fixed shutdown error when closing solr connectors

fix for bad query shortcut hack

fixed wrong used memory display

fixed error cache

fixed usage of media flag when started by automated process

fixed cleaning of index

one more SolrIndexSearcher bugfix

fix for 'AlreadyClosedException: this IndexReader is closed'

fixed shortcut self-reference bug

release 1.7

update of seed list

Update russian translation

update to german locale

fixed a time-out problem in the default servlet which is also a logging
problem because the error log showed the wrong reason (file not found)
instead the actual reason (time-out).

fixed a problem with resource observer; probably coming from uncatched
exceptions within the apache library which appear only in concurrency
environments.

eliminating the word 'Facet' from the interface because it is ugly. If
people do not know what search navigation is, then they also do not know
what a 'facet' is.

enhanced the new link structure graph

better link structure limit calibration

better logging in template engine: shows filename of servlets where
errors in templates occur

moved Edge class out of linkstructure servlet as this does not work on
non-eclipse driven environments (all non-dev cases)

reduced search activity corona strength in network image

move baseurl after parsing orig source in urlproxyservlet
to calculate absolute href links for rewrite from unmodified source.

fix: remove obsolete ref to yacy.home
(use Switchboard instead)

fix for solr query size; too many documents had been retrieved in case
that less than _pagesize_ had been requested.

eleminate obsolete URIMetaDataRow class
by joining it with/into URIMetaDataNode.

introduced new solr field crawldepth_i which records the crawl depth of
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.

added missing servlet html

- added COLS field in GSA response (non-gsa standard by customer
request)
- updated document link in GSA response writer

- added popover to p2p/stealth mode button to explain the peer mode and
privacy issues.
- added popover to first-time use case to explain that specific servlets
are only visible after customization and/or crawl starts

transition.js is also included in bootstrap.js

removed alert.js, modal.js and tooltip.js as these libraries are all
included in bootstrap.js

fix: application link count 
(URIMetadataNode)

free configuration of postprocessing clickdepth maximum depth and time

added tutorial mode - some menu items will only appear if you 'qualify'
for them. Thus, the first-time user will only see four menu items. The
other items will unfold as the user interacts.

fix receiving DHT / parse pultipart
+ another close to fix possible resource leak warning

Update russian translation

added more bootstrap libraries

show host list by default in host browser

make use of detected charset in htmlParser if none is given.

add test case htmlParser.parse  - getCharset
(which fails)

made crawl start pages public since they do not reveal individual
information and they are also not used as servlet to actually start the
crawl (which is Crawler_p.html).

added (again) underline to a tags

Integrated HostBrowser back to administration interface; it can appear
with and without navigation bar.

added template for latest commit

adding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p.html)

fix: cut-off button in WatchWebStructure_p.html
(by header css dd  hight/line-hight)

fix: port in local target adress
& button style 

prevent opening of new files as that could be a cause for the latest
too-many-open-files exception. The old file is just truncated if the
table is cleaned.

added linkscount_i and linksnofollowcount_i to the default solr schema

removed performance settings
- they are incomplete and buggy
- it was not easy to explain
- it did not comply with a KISS strategy
- setting a performance of low priority actually caused crashing of a
peer
- there was nobody who would maintain that functionality

changed the protocol navigator: turned into switch buttons

changed protocol facet to show also all other counts if one facet is
selected

moved host facet up

added typeahead library to re-implement search suggestions for bootstrap
framework.

remove unused private header entries (HeaderFramework)
X_YACY_ORIGINAL_REQUEST_LINE
X_YACY_KEEP_ALIVE_REQUEST_COUNT
CONNECTION_PROP_REQUESTLINE

make transparent proxy handle https-connections:
the implemented handle for connect did not work for me - so lets try the
connectHandler

Update russian translation

- added (previously removed) autocomplete
- removed autocomplete from yacyinteractive.html
- todo: replace old jquery-1.3.2.min.js with latest which is already a
part of bootstrap.css

Team added support for URLs with unicode characters in host part to
blacklist. Punycode is used to handle unicode characters.

corrected line-height in tagcloud

add some InputFileStream close at end of reads
to make sure file is released

limit filetype nav to known extension also on image/media search
- on text search we limit filetype nav already to known extension, apply filter to image search

fix for embedded header: the new css framework always need a headline,
which is the location for the search input frame. The new embedded
header is a clone of the simpleHeader with removed menus and without a
link to the index administration.

Update russian translation

added /date -ranking button

- added text/image/audio/video/app search option to new navigation bar
- changed colors of privacy selector

activated language facet by default

from the administration search bar, do global searches

surrogate import translate dc:language 3-char codes
OAI records often use 3-char language codes, start converting some 3-char lang's to the internal ISO639-1 2-char code

fix: set text color black in select field in table.th in CrawlResults
(default inherite white)

- do not show external link icon on navbar icon
- do not show navbar icon on index.html

restrict write buffer size to a limit

bring api icons to the front

- added stealth button to navigation menu
- more fixes to progress bar

remove check for unused virtual path /currentyacypeer/ 
-  del jqueryheader.template (not used)

ConfigSearchPage.html first adjustment to new layout

added adminAccount switch to ConfigAccounts_p servlet to switch on
protection of all pages; some refactoring as well

added bootstrap-switch to turn checkboxes into toggle switches

added flag to require that all web pages, even such without a "_p"
extension require authorization. (default off)

Update russian translation

reduce number of facets to 10

replaced old navbar with bootstrap pagination

remove AugmentedBrowsing_p.html augmented browsing switch
(has no function in code, previously used in conjuction with http://reflect.ws)

enhanced api menu entries

removed synchronization in embedded solr connection (that was probably
a mistake?)

redesign of menus; usage of bootstrap buttons and dropdown-menus

added dropdown menu in test dashboard

fixed shutdown of solr cores in case that more than one local core is to
be closed (this happens if webgraph is enabled and the index is dumped
using /IndexControlURLs_p.html

use default header for user front-ends and inverted header for
administration front-end

enlarged facet column slightly

removed unused variables and warnings

Update russian translate

removed superfluous import of portalheader.template

remove  AugmentedBrowsingFilters_p.html as none of the settings are used currently
config settings frome the page also removed from yacy.init
augmentation.reflect
augmentation.addDoctype
augmentation.reparse
interaction.overlayinteraction.enabled

fix: remove link to triplestore in Vocabulary_p (triplestore does not longer exist)
- should be investigated in more detail to look for additional implications
Remove "yacyaction" from proxyservlet as it was only needed for removed interaction routines.

tried javascript hack to make statistic divs equal height

remove unused ProxySettings ref. from loader
clean unused whois test code

cleanup: remove unrelated and unused code

upd pom to latest jars

support for multiple sitemaps in robots.txt

remove more unused from legacy server.http
- triggerOnlineAction not used
- useTemplateCache not used

remove redundant url string manipulation in HTTPDProxyHandler
(still used by ProxyServlet)

use servletPath in ProxyServlet instead of fixed name
to allow servlet-mapping via web.xml

fix  DCEntry.getIdentifier 
on successful url parameter

OAI import get multivalued keywords (dc:subject)

allow YaCy to start w/o ssl if JKS init fails

Speed up the ProxyHandler:
simplified cache-storing and make it concurrent in order to free the
clientconnection asap
let other prozesses wait on proxy-access like it was bevore

fix for facets in search rss result: some facets had been missing, all
had missing url properties

fix: to allow unix scripts (bin/*.sh) to allways submit http admin apicalls
using auth via config hash (legacy requirement)

cleanup old reference to HTTPDemon.setAlternativeResolver
optimize .yacyh check in AbstractRemoteHandler

enhanced RowSet re-sort limit for small sets

Update russian translation

Update russian translation

fix: make sure adminAccount changes made via ConfigAccounts_p are effective immediately
force to remove current credentials from knownuser cache

Update russian translation

add index on _val fields, this affects especially title length
an index on fields make search facets on that field possible

Update russian translation

improve OAI-PMH import identifier recognition
- find best fittng identifier (url) by checking all given dc:identifier in record (many entries proviede several identifiers)
  as identifier is currently a multivalued field use "getParams" in preference of splitting the 1st string by ";" 
- add resolve DOI:... identifier via http://dx.doi.org/

- let crawl loader jobs die after 10 seconds without new jobs
- corrected shutdown order t prevent a deadlock during shutdown

Added Index Deletion and Reindex Monitor to German translation

Update russian translation

Added German translation for Config Search and Ranking RWI and Solr

fix: IPv6 INTRANET_PATTERNS for local ip (see http://bugs.yacy.net/view.php?id=378)
requiring following ":" for fc and fd prefix and made pattern match case insesitive
- add some more ipv6 test cases to MultiProtocolURLTest.java

fix: unused / incorrect default username parameter 
(removed setting)

Update russian translation

German translation for Augmented Browser, Parser, Content Control and Vocabulary

Improved Image Collage and WatchWebStructure page display with CSS changes

Added German translation for AccessGrid_p.html, CrawlCheck_p.html, RegexTest.html and env/templates/submenuTargetAnalysis.template

status panel link to set max mem
+url proxy same error text as in transparent

update russian translation

small change to crawler monitor to fit in larger translations

little more verbose proxy 403 error message

(again) full redesign of ConcurrentUpdateSolrConnector to remove
out-of-order transactions regarding add and delete operations. Now all
operations (add and delete) are executed concurrently in-order.

better height for statistic windows

better width for Progress table

fix for switch back to localhost (why did this not work? it worked all
the time)

clean up test sources
 rename to current package names and move to default location

Updated German translation with translated submenu include env/templates/submenuComputation.template

added a forced commit to index deletion to enable synchronized index
updates

make sure that postprocessed documents are overwritten

update russian lang-file and small improve web-interface

a concurrency enhancement which was not used because tests showed worse
indexing speed. I leave the code there since it may be useful in
SolrCloud environments. 

added one more postprocessing low memory check

added a properties file format for the status_p api to support reading
of that information with the java Properties class (very easy for small
clients)

extended the status_p api with disk space information

Merge origin/master

use config value htroot in Jetty init (was hardcoded)
- move htroot exist check from old httpdfilehandler to startup, remove from filehandler and legacy proxyhandler
- use SwitchboardConstant.htroot where appropriate

added short memory protection during postprocessing

removed double concurrency to put Solr documents into the index. The
writings to the solr index are also buffered in
ConcurrentUpdateSolrConnector

fix for switching 'simulate short memory status' and 'Memory Strategy'
thx Thomas

reorg URL test classes add isLocal test with some IPv6 examples
- putting in default location and clean old package names
- add some valid RFC IPv6 sample urls (which don't pass the isLocal test)

edit russian locale

recover from OOM

- better logging for postprocessing
- fixed collection bug in postprocessing

as crawling has a low-cpu demand, we want it to run even if the CPU load
is VERY high. This applies also if the CPU load is high because of
in-cache crawling; in that case we want to experience a high-CPU load as
much as possible

prevent npe

prevent npe

fix: use timeout =  proxy.ClientTimeout in ProxyHandler
(was 10sec fix) see http://bugs.yacy.net/view.php?id=236

Updated German translation and Blacklist_p.html

- introduced a miss cache in ConcurrentUpdateSolrConnector
- better usage of cache
- bugfix for postprocessing

make use of new -config cmd-line parameter in reconfgureYACY.sh
to asure pwd encoding is compatible with DIGEST auth. in future.

changed caching of ConcurrentUpdateSolrConnector: it caches now also the
url along with the load date. While this takes much more memory, it
eliminates database lookups for getURL() requests, which happen equally
often. This speeds up remote solr configurations.

refactoring

removed unused code (HostStat)

introducing ConcurrentUpdateSolrServer for remote solr servers.
Scaling of write buffers and update queue size is made according to
assigned memory.

removed host increment on stored documents from crawler (that was wrong)

edit russian locale

remove unused java imports form yacy.java

add a -config command line parameter e.g. -config "port=9090" "port.ssl=8043"
- useful for remote installation to set any config file property
- multipe parameter can be set at once, on Windows enclose parameter in doublequotes
- special handling   "adminAccount=adminuser:adminpwd"  sets adminusername and md5 encoded admin-pwd

- adjusted windows startbatch to allow command line parameter handling
- remove not needed classpath calculation from startYACY_debug.bat

Finished translation of changed CrawlStartExpert_p.html

fix: allow 4 character admin user name
(was min 5 char)

edit russian locale

removed warning "sun.misc.BASE64Encoder is internal proprietary API and
may be removed in a future release"

Improved German de.lng translation and fixed adresses -> addresses in \htroot\CrawlStartScanner_p.html

better control over close-state of remote solr connections

Edit German de locale and improved translation

fix for
"org.apache.solr.core.SolrCore Too many close [count:-1] on
org.apache.solr.core.SolrCore@51af7c57"
-error

delete fail-docs during postprocessing to prevent that they will appear
again and stay in postprocessing forever.

edit russian locale

add edit-link to PK field of table admin

edit russian locale

edit russian locale

Fixed my error

Edit ru locales

explicit call to optimize to add a expungeDeleted flag

in case that the crawler is running and the search user is the peer
admin, we expect that the user wants to check recently crawled document
to ensure that recent crawl results are inside the search results, we do
a soft commit here.

removed clearURLIndex, which is a stub remaining from the old metadata
database and not needed any more

Added methods in resource observer to calculate the available and the
occupied disc space. These values are also shown on the status page.
The disc space calculation shall be used for a disk-limitation of the
search index.

adopted network image drawing to increased number of peers

upd: Maven pom to dev v1.69

add forgotten solrj-4.6.1

run the cleanup process even when load is high, do postprocessing even
if load > 1 (but < 2) but only if there is enough memory (now: 0.5 GB
RAM available). The memory amount of the postprocessing is the cause
that systems block because they run into a frequent-GC chain which
almost locks the peer. If running with enough memory, the postprocessing
is fast and not damaging to the system.
Because the required RAM of 0.5 GB is never available in default
setting, the postprocessing will not run if the peer is not reconfigured
to use more memory.

replaced guava-15.0 with guava-16.0.1

replaced pdfbox 1.8.3 with 1.8.4

next dev release

upd: Maven pom