YaCy Release 1.80

css skin fix: visited was not set which caused bad colors on new-user
menu design.

Fix for http://mantis.tokeek.de/view.php?id=432

*) replaced all shebangs with more universal versions

remove old description for proxy port settings (Settings_p.html?page=ProxyAccess)
- The options were not current (only port number accepted, which is part of ConfigBasic.html)
- Deleted options and the port number input field from the proxyaccess page.
- joined both transparent proxy setup pages (Settings_Http.inc & Settings_ProxyAccess.inc) in one page
- adjustments to the related/linked pages

added a unlock icon for all protected pages that are unlocked because
the administrator is logged in.

bump to httpclient-4.3.5

update to Jetty 9.2.2

use more predefined Solr query parameter constants
- use CommonParams and DisMaxParams constants
- fix typo in get sort parameter
- getDocumentCountByParams redundant implementation and risk of not optimized call (row parameter unspecified) -> as only used from getCountByQuery removed from interface

simplify rssreader and improve atom feed  link extraction
- type detection (rss/atom) 
    - init type parameter overwritten during parse, parameter obsolete
    - detection by endtag changed to simpler first-tag evaluation
- channel image not used, removed related extra parser handling
    - remove unused code (set/getImage) in rssfeed
- atom link extraction to account for possible multipe link tags
   - spec limits link to one with rel="alternate" or one without rel attribute
     not accounting for the follwing type & hreflang exception yet:

   o  atom:entry elements MUST NOT contain more than one atom:link
      element with a rel attribute value of "alternate" that has the
      same combination of type and hreflang attribute values.

more logic on Solr queries; usage of the query terms in posprocessing,
saving one query for double document detection now per document

better error handling for remote solr queries and exists-checks

update to pdfbox 1.8.6

refactoring (change Metadata name of load time data structure to avoid
confusion with Node data which is also called metadata)

Combine /heuristic search modifier with opensearch configured targets
- with search modifier /heuristic a request is send to all configured opensearch target systems (old /heuristic/blekko modifier not longer valid)
- this allows to use opensearch heuristic on individual search request (in contrast to configuration HEURISTIC_OPENSEARCH=true which sends a osd request on all global searches
- the index.html searchoption text adjusted to be displayed only if option configured
- add Archive-It to predefined systems

added an option to set 'obey nofollow' for links with rel="nofollow"
attribute in the <a> tag for each crawl. This introduces a lot of
changes because it extends the usage of the AnchorURL Object type which
now also has a different toString method that the underlying
DigestURL.toString. It is therefore not advised to use .toString at all
for urls, just just toNormalform(false) instead.

Renamed the transmission cloud to buffer in dispatcher since the name
'cloud' was a bad idea. Changed also the accumulation process for peer
targets so that every dht chunk is not assigned the set of redundant
targets but they are assigned to redundant targets individually. This
enhances the granularity of the target accumulation and should enhance
the efficiency of the process. Finally the dht protocol client was
enriched with the ability to remove the 'accept remote index' flag from
peers or remove peers completely if they do not answer at all.

update to solr 4.9.0

enhanced metadata enrichment for media file type search:
- Web servers may now deliver YaCy-specific http header field with a
title and keywords. The new http header fields are:
X-YaCy-Media-Title - to be used for media (image, audio, video) titles
X-YaCy-Media-Keywords - to be used for media (image, audio, video)
keywords
- both fields are written to document fields title and keywords and are
searched also during image search.
- to make the usage of arbitrary http header fields (including this new
fields) possible in the /api/push_p.json servlet, a new POST argument is
also introduced to push http header fields. The new POST attribute is
named "responseHeader-X" (where X is the counter). It is allowed to use
this attribute as multi-attribute several times, each can be filled with
a http header line.
- see /api/push_p.html for examples

bump to httpclient-4.3.4

update to Jetty 9.2.1

Added filter query options to the ranking servlet /RankingSolr_p.html.
Filter queries are not actually related to ranking, but user requests
have pointed out that specific boost queries to move results to the end
of the result list are not sufficient. Such boost filters may be better
executed as actual filter and therefore such a filter can now be
statically applied to every search request. A typical use could be the
expression "http_unique_b:true AND www_unique_b:true" which uses the
recently introduced fields http_unique_b and www_unique_b which are true
only for one of the alternatives with/without http(s) and with/without
prefix 'www.' in host names.

Added collection navigation:
The collection field (can be filled i.e. in Crawl Start) can be used to
add categories to YaCy index entries. The usage of that field was
restricted to solr searches and post argument filters as implemented in
commit f7571386a3367432648f7cbd6e4744556ff445a7.
This commit extends collections to a full navigation option in the
standard YaCy search interface. The field is not active by default but
can be activated easily in the /ConfigSearchPage_p.html servlet (just
check the 'Collection' facet field). Collections can now be used for (at
least) two purposes:
- to provide search tenants (through post argument collection)
- to provide self-made category navigation
Search requests may now have (independently from switched on or off
collection facet) a "collection:<collection-name>" modifier attached;
firthermore collection names may use disjunctions using the '|' pipe
symbol. For example, this is a valid search request:
www collection:user|proxy

fixed several problems with postprocessing:
- unique-postprocessing was destroying results from other
postprocessings; removed cross-updates as they had been not necessary
- unique-postprocessing did not restrict on same protocol
- inefficient concurrent update cache was redesigned completely
- increased limits for concurrent blocking queues to prevent early
time-out

better handling of ranking parameters and new default values for date
navigation which is done using ranking in solr.

migrated from pdfbox 1.8.4 to 1.8.5. They have a very long bugfix list
for that update:
http://www.apache.org/dist/pdfbox/1.8.5/RELEASE-NOTES.txt

manual merge

migration to Solr 4.8.1
This includes also an update to zookeeper 3.4.6 and a new library that
Solr initializes by default: org.restlet from
http://restlet.com/download/current#release=stable&edition=jse&distribution=zip
which is included in version 2.2.1 from may 6th 2014

fixed generics warnings for generic array instantiation that appeared
after migration to Java 7

update to Jetty 9
besides adjustments in code it makes the servlet settings in web.xml significant.
This applies to solr, gsa and proxy servlet. There is no longer a default setup in code during init (as jetty 9 checks for double definition).

fix for http://forum.yacy-websuche.de/viewtopic.php?p=30720#p30720

fix for favicon

changed error message, see http://mantis.tokeek.de/view.php?id=439

fix for http://mantis.tokeek.de/view.php?id=467

fix NPE in image search

enhanced image search: bugfixes and performance enhancements

fixes and speed updates for search process

fixed text in startup tray icon and added shutdown icon during shutdown

npe fix for non-tray users

very bad language detection hack fix hack

fixed wrong submenu title

small fix to the welcome message

another fix to de locale which fixes a html tag glitch

fix for translation mistakes

fix for de locale translation mistake

bugfix to latest commit

addon to latest fix

fix for browsing of file paths in Index Browser

concurrency bugfix

fix for possible interruption of concurrent queries

fix for subpath crawl filter

fix for termination of all crawls

fix for crawler attribute maxdompages

npe fix in crawler

toString fix

fix for image alt attachment to AnchorURLs in html parser.

This hopefully fixes http://mantis.tokeek.de/view.php?id=424

fix for malformed hostpath names in crawl balancer

toString() fix

fix for file urls

fix localhost link to opensearchdescription.xml

fixed target_alt_t in webgraph

fix for condition for writing the webgraph

small bugfixes

fix button (clear list) text color in CrawlResults

added debugging of filter queries in thread dump thread names

another fix for unique field computation

fix for processing of noindex flag in http header

fixed bugs in canonical, robots and title/description unique calculation

fix for bad URL decoding

fixed double content postprocessing

fix for long periods in timeline

refactoring of AccessTracker events & timeline fix

fixed external link

enhanced debugging

bugfix for image search

increased fixed limit for http POST request sizes to 100MB

fixed and enhanced Base64 (en)coder (again)

Improvements and bugfixes for recording actions of blacklist API.

fix for broken json

fix for crawl start check

NPE fix for postprocessing without term index

fix for bad String conversion

fixed webgraph postprocessing and status display in Crawler_p servlet

fixed dependency in debian package on java 7

fix appearance of image search thumbnails.

debug code for index distribution testing

fixed bug in test function

fix for latest generic warning fix

fix: allow enable of CrawlStartExpert.html #file

fix xml validation error on defaults/web.xml

Release 1.80

updated seedlist

replaced warning with info

more enhancements to image search in case that a restriction to a single
domain is done

hack to get more results when doing a remote site search

modifiy description for Field-Reindex to act only on local index
http://mantis.tokeek.de/view.php?id=279

increaesed ability ot search for all images on a single server within
the p2p remote search

better contentdom abstraction

changed default value for maximum number of connections to 50

automatically add a wild card if only a search on a single domain is
done. This makes it possible to search all documents on a single domain
even if no search word is given. This is in particular interesting when
searching for all images on a single domain.

removed www and welcome servlet, these had been demo servlets and are
not needed any more

implement a forward to remote peer link in P2P Network list
Most links in Network.html are only available with transparent proxy = on, which is switched off by default,
to make the provided links useable in default setup a small forward servlet added (goto_p.java),
which takes the peer hash as parameter and forwards to current public ip (optional with path= parameter).
The servlet is protected ( _p ending) to assure forwarding works only for authorized YaCy users.

add recommended link "self" to atom feed output

limit OAI import urn resolver try for Deutsche National Library
The resolver service of National Library uses name space nbn, limit use of nbn-resolving.de accordingly to urn:nbn:
- add resolver for rfc's

added a script which retrieves the public ip of the local peer and
prints it out to the terminal using the new servlet /yacy/seedlist.xml.
just call:
bin/myip.sh

Added the option to retrieve only the own seed or a seleceted seed
(selected by peer-hash) in the /yacy/seedlist.[json/xml] api.
Added also a xml-version of the servlet.
The own seed can be retrieved i.e. with
http://localhost:8090/yacy/seedlist.xml?my=
http://localhost:8090/yacy/seedlist.json?my=
and any other peer can be selected with
http://localhost:8090/yacy/seedlist.xml?id=<peerhash>
http://localhost:8090/yacy/seedlist.json?id=<peerhash>

- remove repositoryPath post from ConfigBasic (obsolete)
- remove static snippetComputationTime from ResultEntry (not used)

changed link to new tutorial repository (yes, Youtube..). The link does
not point to youtube directly to prevent that the referer to the peer
address is given to youtube. Instead, a forwarder address at yacy.net is
used to redirect to the tutorial repository (and can be changed later).

Update russian translation

- skip html template specific servlet post variables (show_xxx) for feeds,
- add <updated> (in required format) to atom feed

adding totalResults and id to atom feed output

Update russian translation

Update russian translation

use configured search items per page if not specified in  post
- remove verify=cacheonly from admin screen search box to use the configured values
  (otherwise definition overwrites configured behavior and is used for following searches what might give unexpected/confusing different results compared to using /yacysearch )

Update russian translation

using htroot/api/blacklists as source folder: removed package
declaration of some classes in that folder

changed translation of submenuCustomization.template to submenuDesign.template
- removed obsolete scroogle translation

replaced font for boot tray icon with image and added some more images
for further tray icon displays

speed-up of start process if remote DNS waits for timeout

added an animation of the tray icon during the boot phase of YaCy.
Additionally, there is a tooltip and a new headline at the tray menu
which states the current booting status.

adding the buffer size is not wrong but may cause confusing information
when the buffer is cleaned after a buffer flush which is not then
available in Solr since that is waiting for a commit. In such cases the
counter would run backwards which is prevented by ignoring the buffer
size.

changed strategy to count the number of documents: get the max of
solr+buffer and the hit cache. This shall help during first crawls to
see a running document counter even if there was no commit meanwhile to
solr. To support that strategy, the hit cache must be written earlier.

set the correct fail time in error docs

Update russian translation

Update russian translation

removing the [] for ipv6 addresses may be a bad idea..

enhanced the apk parser (up to beeing production-ready).
The parser is not yet activated and will be after the next release step.

removed solr warning during startup

changed method to write the citation index: do not catch up references
during document parsing; instead use the same references that would also
be written into the webgraph. That should cause that the webgraph and
the citation index express the exact same semantic.

fixed localhost authorization and replaced the adminRealm with an info
string which is visible in the browser. That makes it possible that the
browser instructs the user how to change a forgotten admin password
(during runtime).

moved index administration up ahead of system administration to put more
importance on it. People should not feel that it is more important to
tweak any settings (which may break things) than to look into the index.

moved Table administration and Busy Queues Config out of mini-submenu of
advanced settings to a top-menu entry. Moved the advanced setting to a
less prominent place of the submenu. Removed the table administration
from target analysis submenu because it appeared double, the table
administration is now the default in the system administration.
Sorry for inconvenience if i constantly move menues around, but this
makes just more sense and YaCy is still not finished :)

moved cookie monitoring to the network monitoring submenu

enhanced date-ordered ranking

switched position of API steering and content semantic

fix Querygoal.parse exception on +/-null-term
covers http://mantis.tokeek.de/view.php?id=452

exclude nav-header/footer  in ServerScannerList.html?embedded
fixes display of header in yacyinteractive.html

remove unused legacy procedure from httpserver
(deleted  generateSocketAddress(port) )

bugfix for wrong lock display, see
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5321&p=30484#p30484

update to kaskelix

added hint to the search api examples to remind the user that these
interfaces are examples and not actual variants of the current search

added hint to the regular expression tester

changed style of info-icon to be similar to bootstrap.css glyphicons

another translation typo

add rss version to api/feed.rss
IE11 reports error without

do not remove the index deletion option from the IndexControlURLs_p.html
servlet after a deletion happend, instead show but disable the option
when the index is empty.

added a confirmation dialog to complete index deletion

fix html error occurred by missing symbol in de localization in
IndexDeletion_p.html

enable Crawler_p servlet if a rss feed or a wiki dump import was
submitted.

do not push noindex errors into log if in intranet mode. noindex
attributes are attached to artificial constructed index.html files which
list directories. Such files are naturally rejected by the crawler and
should not appear in the error log because these files are part of the
construction of file crawlers and confuse users if they see them in the
error log.

Improved UPnP. (still not perfect)

*) set HTTPS port if enabled
*) improved data structures (may not be final)
*) moved UPnP to own package

removed the HTDOCS repository concept because the concept to host files
on the YaCy http server is obsolete; YaCy can index file:// and smb://
paths

renamed Host Browser to Index Browser (gives a better hint what it
actually does)

let NETWORK_WHITELIST take precedence over NETWORK_BLACKLIST
this makes it easier to config exception (for private networks),
like   blacklist= .*
        whitelist= 10\..*,127\..* .....     allows only listed ip pattern

change seedfile upload ip check to allow intranet ip in intranet mode
- this allows to setup a principal peer in intranet environment

update target url for yacy-portalsearch.html to search.yacy.net (peer yacportalsearch2014)
(old www.yacy-suche.de not reacheable)

- update javadoc Seed.getIP
- default mySeed.ip to hostip in SeedDB.initMySeed() if Intranetmode
this allows to become senior status in intranet hosted search network with view peers,
otherwise peer would stay junior because of default init with loopback ip as public (dna) ip.

in IntranetMode allow intranet hosted seedlist with Network_Domain "any"
- so far intranet seedlist hosts are always denied but need to be allowed in intranet mode

germal locale correction

make sure that the postprocessing background thread never dies by any
exception

seedUploadUrl config is lost on restart if no publish event occured
-add a saveMySeed() on uploadurl changes (to keep url setting without retyping even if network down)

added 0x40 to image of lockopen-gif image palette (light grey)

changed lock/unlock image design

set default "Search Interfaces"."Solr RSS/Opensearch" query to show latest 10 addition to index

update Netbeans IDE classpath

update to commons-compress-1.8.1
(bugfix release)

reduce mem buffer overhead for gap files during  r/w
(they are typically small compared to idx allowing to use smaller buffersize -> set to 16k records)

allow single quote " ' " in query
see http://mantis.tokeek.de/view.php?id=379
-add QueryGoal test case for this

fix: hanging text in input fields of WatchWebStructure_p.html
in IE11

snippet computation update

reverted template from latest cherry-picked commit

updated some french translations

fill solr rss writer dc:subject tag with keyword content

updated fr traduction of ConfigBasic.html and header.template + fixed some accents in fr.lng

removed warnings

add type attribute to atom feed <link> tag
(for /yacysearch.atom)

reverted .classpath from cherry-picked
https://gitorious.org/yacy/heronchatoyants-rc1/commit/ce1fd1e90170f6de4ee3fce1c6acbd2d8d7ba9ad

updated the traduction of Status_p.inc in the french locale

fix test case MultiProtocolURL.toString()
(only allowed on AnchorURL)

eliminate lat/lon type conversion in osm
(define as double)

reduce doublecontent check document (prevent out of memory)
see http://mantis.tokeek.de/view.php?id=437

test result (concurrency=7)
2000 docs = eom always
1000 docs = eom always
100 docs = eom never

chosen -> 200 docs (eom not encountered during test with 1GB mem setting)

catch NPE during crawl (OAI import)
- condenseDocument mime=null (allowed)
- collectionconfiguration responseheader = null (allowed) 

center searchevent lines on network graph
(PerformanceSearch_p.html)

increased default logging line count to max

reducing the concurrent query stack size and reduced concurrency of
postprocessing to avoid OOM situations

bugfix for crawler double-check: if an url is redirected, the
redirect-target was not double-checked. This is now done by replacing
the redirect-URL on the crawl queue again (where it is double-checked)

added hint to port forwarding videos

fix ConfigAccounts del user with uppercase letter in name
(usernames are case sensitive, userdb.delete used toLower) 

typo

added concurrency to postprocess rewrite process

log ppm instead of docs/second

set process type of web graph according to fields as defined in the
schema

log postprocessing constraints in case that postprocessing is not
performed

less logging

better solr query logging to detect unnecessary sort requests for more
performance profiling

no sorting if http/www unique fields are not demanded (makes query
faster) and some code restrucuring

more retries, less sleep

fix "Ljava.lang.String" in crawl queue anchor name
(e.g. IndexCreateQueues_p.html?stack=LOCAL with images in queue)

catch IndexControlRWIs_p error if RWI not connected

added new classes which shall reduce call overhead to Solr (stub)

adjust rss lat/lon to double
(common format across other classes)

removed unused images from webgraph edge computation

target linktexts must be string to enable search facets on these fields

do not store failure of loading of robots.txt into the index as a fail
document

- update Maven pom
- add ppt parser test case

Replaced old UPNP library with Weupnp. UPNP should
 work now, at least it does on my network. UPNP code in YaCy can still
be improved though (see TODO comment: make port on gateway configurable
or find free one).

*) removed old code
*) added new lib
*) changed code to work with new lib

reverted commit f94c91315bce0fb5a328ce7204a9066ff457e9e3 because the
webgraph has not enough performance for that

yacystats is gone :(

enhanced snippet computation and test method in ViewFile

move bootstrap peers test systems to its test class
var assignment not needed  elsewhere.

fix type in .init description
http://mantis.tokeek.de/view.php?id=430

set .ini default boost fields to same as assigned by button "reset to default"
(in RankingSolr_p)
- fix typo http://mantis.tokeek.de/view.php?id=430

fix for the usage of ready-prepared solr queries, some queries are
formulated as edismax query but this was not set as query attribut. The
defType=edismax property needs a qf-field, so this was added as well. Do
not remove that field again! This fixes also a problem with title-unique
computation.

if the webgraph is used, then use it also for reference computation to
avoid contradictions with references_i in the collection index.

added a snippet test function in ViewFile: you can now search for a
specific word on the document; the servlet returns the snippet in the
same way as it would be shown in a search result.

no idea why the words max was an ordered one. This change increaes speed
dunring document processin a bit

Merge origin/master

fix double &query parameter (index.html)
?query=word&query= 

Updated commandline reconfiguration tool.

*) fixed "set HTTP port" (root cause was sloppy implementation of method
which gets values from config file)
*) added "set HTTPS port"

changed handling of crawl profile field crawlingIfOlder: this should be
filled with the date, when the url is recognized as to be outdated. That
field was partly misinterpreted and the time interval was filled in. In
case that all the urls which are in the index shall be treated as
outdated, the field is filled now with Long.MAX_VALUE because then all
crawl dates are before that date and therefore outdated.

improve logexception info
- log a message or class name insted of msgtxt "null"

fix bookmark.rss
- channel end tag postion
- link with html entity

reverted toString() change in AnchorURL to prevent mistakenly used
toString(). This fixes also the update link bug.

Add link to RemoteCrawl_p
suggestion http://mantis.tokeek.de/view.php?id=277

Add Solr default query fields (&qf) to select servlet
according to the ranking profiles boost fields defined by the peer (if df/qf is not specified in query).
This allows for pretty simple queries ( q=word) without the need to know about the specific index configuration.
Making sure all relevant fields (as determined by the index owner) are searched, still maintaining the option to query specific fields
and does not relay on the duplication of text to text_t.
- add author to reset-default boost fields (support results for author nav)

prevent array out of bound exception on getRankingProfile(x)
on faulty &profileNr=  query parameter 

fix input boostfield factor of 0.0 in RankingSolr
- input was accepted and stored but not editeable (added check factor >0.0 during edit)
- make use of some more predefined solr constants

remove unused static (RSS)search from protocol

use CRLF in generated bat command scripts for windows
- for easier viewing with standard viewers

do not write CR values to webgraph if no CR values are computed

added option to delete documents from the webgraph

added SolrCloud access mode and configuration

enhanced postprocessing status report

Update russian translation

fix GSA filter query assignment
- use more parameter constants

removed exist() retrieval functions from error cache and replaced it
with metadata retrieval from connectors directly. This should cause
better usage of the cache. Automatically increase the metadata cache if
more memory is available.

cleanup of checkAcceptanceInitially in CrawlStacker, should avoid
double-calling of solr

reverted filter query hack, it did not work

reduced number of solr queries during crawling

enhanced solr queries for single document extraction

testing filter queries instead of queries to retrieve documents by id

removed more unnecessary exist-checks in ErrorCache

don't do a double check to solr for failed documents if they are not
written to solr

enhanced crawler host round-robin strategy

catch input file not found in Mediawiki import

cleanup older unused classes

move startup time from old serverCore to switchboard (most used here)
to make servercore eventually obsolete.

remove unused localSearchThread from SearchEvent

add local ip to AbstractRemoteHandler local hostname cache

added linkScraperParser, a parser which ignores the text like the
generic parser but extracts links like the htmlParser. This should be
used for ASCII documents without known text format annotation like
source code files or json documents. Probably also good for xml files
without known schema.

fix seek error for 0 file size records file
by add extra check for file size = 0 in cleanlast()
- (http://mantis.tokeek.de/view.php?id=411)

add test case for Records (used in HostBalancer)
- simulating seek error (http://mantis.tokeek.de/view.php?id=411)

fix Crawler HostQueue release of  stackfile
- close stackfile inputstream at end of ChunkIterator
This should solve startup delay while unfinished crawl jobs exist (maybe also too many open file situation)

fix NPE in Condenser,
discovered by calling IndexControlRWI, "Word Deletion" with "for every resolvable and deleted URL reference"

update to JUnit 4.11
- fix build.xml -> parserTest error on Windows due to javac encoding

adding test for HostQueue crawl stack
- simulating problem with zero length stack file (but not fixing it)
- adding test data clean to maven pom

make search result also avail. as atom feed via /yacysearch.atom
- fix logo in rss feed 

timeline performance update

reverted latest change for reindexing because that works actually only
for internal Solr indexes. This is mainly caused by the fact that an
external Solr may be also a SolrCloud which do not support LukeRequests,
which are needed to request the old Schema.

remove check for local solr. This check was made during a time when Solr
was optional and another alternative metadata store was available. Since
that store is now removed, Solr is always available (internally or
externally)

fix NPE in solr rss where snippet contains only the title text
and adjusted xslt, for solr snippets (&hl=true) to decode the xml encoded html <b> tag by adding disable-output-escaping
(still open item description may be double as dc: tag and rss.description tag)

update of commons-io and slf4j-api (as preparation for Solr 4.9.0)

added test case for TextSnippet,
removed obsolete/unused parameter and reference to MediaSnippet

no timeout for large reference collections

extract author and keywords in .doc and .ppt parser

enable proper Author navigator
- author facet is based on omitted author_sxt field
- adjust to make author nav available on exist of author field but keep using author_sxt to construct the facet (why!?)
- add check for querymodifier author in searchevent

refactoring

added apkParser stub (work in progress)

added access tracker log reader stub

adjust mergeDocument after parsing to
- preserve charset and languages
- fix merge of author

fix AbstractParser.singleList not adding null strings
- prevents null titles in oo... parser  (as detected by ParserTest)
- correct ParserTest dc_description check (dc_description allowed to return 0 length array)

write <em>-tagged texts also into the bold_txt field

harmonize use of style for "delete" button
- apply the monstly used btn-danger class 

add a link to recrawl index entry to metadata html page
- to allow manually renew index content for this url (e.g. in case it is a remote search result with metadata only)
- use simply a  QuickCrawlLink_p javascript snippet (minimalistic 1st solution)

added 'synchronous' and 'commit' flags in push api

using concurrency to do base64 encoding in file POST commands

enhanced ASCII byte[] -> String conversion

move page navigation below results (image search)
force page navigation to be displayed below results in image search for any number of displayed images instead to be displayed to the right of last image.

enhanced base64 encoding

removed superfluous date method

removed superfluous assert

ignore the api javadoc directory in git commits

added a push api to make it possible to upload files directly without
crawling to the YaCy indexer. Files are uploaded using POST multipart
requests; multiple file uploads are possible as well. Each file has
attached the file date and mime type which is used to get the right
parser for the submitted data. Also an url is submitted which is
assigned to the document.
The CrawlSwitchboard has a new option for default Crawl Profiles which
are assigned dynamically from the new push interface.

re-implementation of file push option in the context of POST http
requests. The internal representation of post-arguments is String and
therefore not appropriate for byte[] object as submitted by file pushes.
Therefore all pushed files are encoded to base64 _after_ uploading with
an http form (you do not need to do that encoding yourself) to hand-over
the byte[] as string in the post argument.
Servlets which read such files must decode the base64 data to get the
original byte[] array.
This is considered as a temporary solution for file uploads and a proper
implementations would need to consider all attributes as handed over as
Objects with either String or byte[] Object instances. This would be a
major code change and is not done at this time here now. The feature was
submitted to realize a feature as pushed with the next commit.

refactoring

crawling of filenames with + fails due to url decoding
modified UTF8.decodeURL to apply x-www-form-urlencoded ( space -> + ) to the query part of the url only.

optimize pdfParser
skip starting reader thread if all content already read

fix pdfParser not closed warning from pdfbox
for encrypted pdf on exit due to missing permission to extract

fix unresolved pattern in yacysearch.rss title
and rss xml error due to html & encoding in url entries 

Crawler hostbalancer does not delete finished queue files,
use alternative delete to fight the sympthom (and fix deletion of host dirs on startup)
Root cause (which class holds a lock on .stack) not found.
http://mantis.tokeek.de/view.php?id=404

in case that the network is switched to any other than freeworld, RWIs
are disabled. This is a temporary fix. There must be a better way to
determine if RWIs are to be switched on or of.

doing index deletions using a get string which makes it easier to
copy-paste deletion examples (see: #EuGH :( )

added option to prefer http over https in unique-protocol ranking

removed check on protocol for double content (user request)

add links to port config in status panel
- pom upd to match javadoc location

changed javadoc publishing path from 'api' to 'javadoc' because there
are also other APIs in in YaCy.

fix logging settings
- add missing .level
- remove obsolete jena settings
- set default level=INFO to prevent debug logging of not explicite specified classes

make https port setting changeable via front end somewhere
(chosen Http Networking page /Settings_p.html?page=http )

added option to configure a custom user agent within allip networks

- fix typo in Message_p (defaultpath)
- use more existing switchboardconstants for getproperties
- replace depriciated call defaultservlet

made number of open files in crawler configurable and increased default
maximum number of open files from 100 to 1000. This number can be
changed with the attribut crawler.onDemandLimit

Update russian translation

Changed Windows Firewall Rules to just honor the default Port 8090, but
not use any programm-path.
This should match more installations in different paths and also running
YaCy as service (prunsrv).

This commit was contributed and tested on Windows7 by René.

a try to fix the mixed up terms 'Active' -> 'Senior' and 'Passive' ->
'Junior'

fix crawl of file:// , also http://mantis.tokeek.de/view.php?id=149
local files can be crawled (intranet mode) url parsing fixed according to  RFC 1738 (for unix and windows)
for win like file:///c:/tmp   or file://localhost/c:/tmp
for linux like file:///tmp  or file://localhost/tmp
Host is ignored and path must be absolute

changed debian dependency to openjdk-7-jre-headless

added new fields http_unique_b and www_unique_b which can be used for
ranking to prefer urls containing a www subdomain or using the https
protocol

make config search page layout closer to actual page appearance

- upd pom to Solr 4.8.1 and latest jar updates
- upd nsis java autodownload package to jre 7u55

Monitoring and limit connection-count for Jetty

upgraded poi library (office document format parser) from 3.9 to 3.10

switch on core.service.rwi when switching back from portal mode to p2p
mode

upgraded bouncy caste libraries (needed for encrypted pdfs, dependency
in pdfbox) to 1.46
removed the activation.jar library; I don't know which other library
depends on it.

tried to make last_modified recognition smarter

'Last-Modified', not 'Last-modified' according to
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

fix for operator precedence bug (cast binds stronger than bitwise AND)
in peer hash hashing. This should not change anything if java casts long
to int by masking with 0xFFFFFFFFL but you never know. The important
thing is, that the hashCode() should not return numbers that have the
same order as the hash code order because hashing of seeds is used to
remove the order in some places.

Revert "manual merge"

This reverts commit 3bfab8566c30df5a928ec6c7d3736f8e5d2eb4ca.

avoid erros from ConnectHandler, correction for #6d16fa9

updated bootstrap seed list

Improved blacklist XML/JSON API.

lower Segment-size (hope to get Segments of 10GB)
see:
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5216&p=30036#p30034

fix: load_Rss remove feed to many parameter for get
use form post methode

"Use remote proxy for yacy" has no function, remove option and related config item
see/fix bug http://mantis.tokeek.de/view.php?id=23
http://mantis.tokeek.de/view.php?id=189

fix: CrawlStartExpert.html # From File with missing filename
- crawlName must not be empty
- crawlingFile must not be empty

cleanup obsolete and not used  serverswitch Authentify code
as auth is mostly delegated to Jetty container.

move unnecessary nested else out of condition 

removed warnings

removed warnings

another circle computation enhancement

switched menu position of ConfigPortal.html and ConfigSearchBox.html

Exclude =  from percent-encoding in MultiProtocolURL
fix http://mantis.tokeek.de/view.php?id=185 and http://mantis.tokeek.de/view.php?id=280

fix truncated urls containing ","
adressing http://mantis.tokeek.de/view.php?id=58

Exclude comma from percent-encoding in MultiProtocolURL (see  RFC 1738 2.2  and  RFC 3986 2.2)

enhanced circle tool initialization

automatically clear graphics cache

enhanced circle computation time and memory footprint

improve IndexDeletion by query
adding transparently text_t as pseudo default search field if no fieldname (no  : ) is included.
adressing bug report  http://mantis.tokeek.de/view.php?id=274

assign serialVersionUID for proxyservlet, too.

set serialVersionUID on urlproxyservlet to skip compiler warning
- remove commented out code

remove redundant servlet-api

respect "index media" switched off in CrawlStartExpert.html
fix http://mantis.tokeek.de/view.php?id=64

Merge origin/master

switch pom to v1.73 and java 1.7

added a peername=<peername> property in the seedlist API

next development step: migration to java 1.7
This includes also a small code change to test generic type inference, a
java 1.7 feature