YaCy Release 1.2

added solr faceted search support to YaCy search results
added solr highlighting / YaCy snippets to YaCy search results
- facets are now much more complete
- facets are computed and searched much faster
- snippet computation is done by solr if solr knows the snippet

added the visualization of error-urls to host browser
- only visible for admins
- a faceted search generates a huge list for all hosts in the host list
- the faceted search algorithms had to be modified for that
- within the browsing of the directory path, the error cause is written
to the url which is presented as error-url
- the errors are also accumulated for directory sums

update to web interface structure

renovated the way how search results are count. should be correct now...

update to HostBrowser:
- time-out after 3 seconds to speed up display (may be incomplete)
- showing also all links from the balancer queue in the host list (after
the '/') and in the result browser view with tag 'loading'

migration to solr 4.0.0

tried to clean up the search process mess

fixed a problem with local search from solr results: now all results
from solr are shown (again)

- added a delete button in host browser to delete a complete subpath
- removed storage of default collection name - default is now "user"
- made stacking of crawl start points concurrently

replaced the custom robots.txt loader by the standard http loader

- removed unnecessary synchronized and deadlock in crawler
- removed problem with monitoring object on Balancer.wait
- added missing user agent settings

update to Balancer algorithm:
- create a load list from the current list of known hosts
- do not create this list for each Balancer.pop access
- create the list from those hosts which have a zero-waiting time
- select 1/3 from that list which have the most urls waiting
- get hosts from the wainting list in random order
- fixes for some delta-time computations
- always load all urls from hosts which have never been loaded before

- added a method for the RasterPlotter to draw arrow endings to lines
- replaced the dot in the NetworkGraph with arrows
- enhanced the image drawing speed using pre-computed color values
- added more attention for OOM cases during very large image painting

removed warnings

Refactoring and redesign of data architecture to make URIMetadataRow
superfluous. The target is to make a solr document as the core of YaCy
documents which would cause that many conversions can be removed. On the
way to this target the Equivalence of URIMetadataRow and URIMetadataNode
had to be removed to expose the usage of the old URIMetadataRow data
structure.
This refactoring already removes unneccessary conversions and should
make memory usage during indexing lower.

removed hack which translated Solr documents to virtual RWI entries
which had been then mixed with remote RWIs. Now these Solr documents are
feeded into the result set as they appear during local and remote
search. That makes the search much faster.

- removed dependencies from URIMetadataRow and made direct access to
URIMetadataNode which creates the opportunity to access Solr objects
directly and use their information richness
- lazy initialization of the URIMetadataNode object - should cause less
computation and memory usage during search.
- removed dead code

enhanced the HostBrowser:
- showing also outbound links to other domains if there are any
- the outbound links browser shows also the link structure image
- showing even inbound links if the web structure graph has information
about that
- removed the left menu and made the HostBrowser a part of the top menu
for search
- moved the file search also to the top menu
- added hover information in the HostBrowser to explain what the click
means
- because the HostBrowser also links to the Metadata viewer ViewFile,
there should be a button to switch back to the HostBrowser: added that
also.

- enhanced generation of url objects
- enhanced computation of link structure graphics
- enhanced collection of data for link structures

removed the option to prevent removal of & parts inside of the
MultiProtocolURI during normalform computation because that should
always be done and also be done during initialization of the
MultiProtocolURI Object. The new normalform method takes only one
argument which should be 'true' unless you know exactly what you are
doing.

since the solr index is now used for all pages that are indexed locally,
there is no need for the RWI index if the index is not transfered to
another peer. Therefore the creation of RWI index data is now suppressed
if DHT is disabled. This applies for all intranet and portal mode
configurations, but not for public robinson modes. A robinson may switch
back to public mode and then transmit its data. That means if someone
wants to switch never to DHT mode, it would be more appropriate to
choose the portal mode.

code cleanup: removed unised methods and made more methods and objects
private

- redesign of solr query construction
- fix for solr boosts and location search
- fix for number of search results in local search

clean-up: removed unused methods in kelondro

Merge remote branch 'origin/master'

Added a parser for audio file tags (e.g. ID3 tags for MP3 files) based
on the jaudiotagger library. The parser is disabled by default as it
needs to store temporary files for non file:// protocols, which might be
disliked. For your local MP3-collection it loads nicely Artist,
Title, Album etc. from the audio files meta data.

added a synonyms_t field to solr and a process to read synonym files.
This can be used to add another stemming to solr using stemming files
that are expressed as synonyms for grammatical alternatives. The
synonym/stemming files must have the following form:
- each line is a comma-separated list of synonyms
- the list of synonyms may be enclosed with {} (like the GSA synonyms
file)
- the file may contain comments which are lines starting with a '#'
The synonym file(s) must be placed in DATA/DICTIONARIES/synonyms/ and
are activated by default whenever a synonym file is in place.
Then, for each word that is found in a document all synonyms are added
to a long text field which is stored into synonyms_t. Processes using
the synonyms must query with that field as optional matcher.

added new Host Browser to main menu:
this new search interface is something completely new for search, but
completely common on desktops: browser a web space like one would browse
a file system in a file browser. The file listing is created using the
search index and a faceted restriction to specific domains.

extended solr connector with a method to retrieve a single facet.

- removed ip_s from default profile since that needs a DNS lookup to
create an document entry. This makes remote search much slower.
- removed synchronization of add method if ip_s is activated to prevent
that a user configuration causes bad behavior. The disadvantage of that
is, that a index dump can cause data loss if an indexing is running
during index dump
- catched more exceptions and more NPE
- better abstraction in MirrorSolrConnector
- slight performance enhancement when only the index count is requested
(rows=0 is sufficient to get a total count)

refactoring

refactoring

refactoring

refactoring

removed more dependencies in cora from kelondro

removed kelondro dependencies from cora

- added the possibility to have not one but a list of crawl start urls
- the list of urls is entered in the expert crawl start in a textfield;
the one-line input field was replaced with a text box
- start urls can also be given in one single line where the urls are
separated by a '|'-character
- as an effect, the crawl profile cannot carry a single start url for
identificaton because it is possible to have more. Therefore the url was
removed from the crawl profile
- this affect all servlets which display a crawl profile: removed the
url field from all there servlets
- to work consistently with several start urls and the other crawl
starts which computed crawl start url lists from sitelists or sitemaps,
the crawl start servlet was restructured completely
- new rules for must-match patterns were created to make it possible
that site crawl starts also work with several crawl starts at once

- updated lucene libraries to 3.6.1
- added lucene-grouping which enables faceted search; try this:
http://localhost:8090/solr/select?q=*:*&start=0&rows=3&facet=true&facet.field=host_s

Merge remote-tracking branch 'origin/master'

Conflicts:
	htroot/api/ymarks/import_ymark.java
	source/de/anomic/data/ymark/YMarkEntry.java
	source/de/anomic/data/ymark/YMarkTables.java

- added dmoz RDF dump importer
- added indexing to Tables columns to support larger bookmark
collections
- added RDF output (HTTP) for public bookmarks at /YMarks.rdf
- YMarkRDF also provides a Jena RDF Model as "internal" API
- various other changes/fixes for YMarks (mainly backend)

small style changes

added new button design to more buttons

added a collection attribute to crawls and searches:
- a solr field collection_sxt can be used to store a set of crawl tags
- when this field is activated, a crawl tag can be assigned when crawls
are started
- the content of the collection field can be comma-separated, all of
them are assigned to the documents when they are indexed as result of
such a crawl start
- a search result can be drilled down to a specific collection; this is
currently only available in the solr interface and also in the gsa
interface using the 'site' option
- this adds a mandatory field for gsa queries (the google api demands
that field all the time)

- extended the solr interface by a references-by-word-count method
- reduced danger that a non-existing RWI database causes NPEs
- added Solr queries to did-you-mean: this makes it possible that our
did-you-mean algorithm works together with only Solr and without RWIs

- added new solr fields:
title_count_i, title_chars_val, title_words_val
description_count_i, description_chars_val, description_words_val
- added many asserts to ensure data type correctness from YaCy to Solr
and vice versa
- made many fixes according to new findings from these asserts (!)

add content control features for custom filter lists

- added a solr type definition verifier
- fixed type definition found by the verifier
- added multivalue-string fields for solr with extension 'sxt'
- added multivalue-integer fields for solr with extension 'val'
- renamed some solr attributes from txt to sxt
- changed solr query line to an explicit AND/OR structure
- added a country code second level domain list to Domains class; with
parser
- added a host string parser to get domain class name, country-code
second-level domain and subdomain out of it
- removed old coordinate attributes

- added faceted drill-down for host and geolocation to solr queries
- added a new geolocation field to index schema, the old values are
migrated if possible

fixed media search

fixed a problem with non-terminating crawls

fix to ftp client

fix for filetype naviagtor

bugfixes for crawler

fixed npe for surrogate import

fixed wrong order of result count values

fix for some interface problems

fixed filetype modified for media types in text search

fixed getSize() which can use the cache size while the crawl is running

fix for host browser

fix for highlighting in gsa search

fixed more getSolrFieldName usages

fix for file parser problem

added an exception catch

fixed clear scripts

fix for crawl start filter

fixed interpretation of directDocByURL attribute during crawl start

fix for ViewFile

fix for portal mode

fixes to crawl profiles

another fix to location search

fix for location search query encoding

some more after-refactoring fixes

fix for debian package installation (caused by refactoring)

fix for java.lang.RuntimeException: TableColumnIndex not available...

fixed bad output in stopYACY.sh

fix for no depth limit default value

fixed size parsing in RSS message parser (for YaCy size parameter)

fix for success query counter

fixed a bug with size_i field usage

*) fix for http://www.yacy-forum.org/viewtopic.php?f=2&t=759

fix for images_withalt

added more patches to work without RWI data structure

fix path to env/grafics to display api icon on meta data page

release 1.2

added new submenu 'Target Analysis' with three servlets which are useful
to analyse the target servers: robots.txt table, mass target analysis
and a regex tester

do the commit anyway before calling a search interface

using a better file name

removed warnings, removed too-fast pausing of crawls

added matching of path to query pattern

update to search result logging (this was a remaining issue from the
solr 4.0.0 migration)

better colors for host browser and corrected document count

update to HostBrowser

removed location search because it is only working in special cases

more logging

automatically delete entries from the crawl profile list if crawl is
terminated.

added information about the reason of pausing of crawls

added more thread-renaiming for search processes

set the thread name during solr queries to the solr query to get better
debugging options

when a new crawl is started, delete all entries about error-urls for
crawl-start domains

added a hack which makes the HostBrowser more performant when the given
host has a lot of urls. If the number of urls is > 1000, then the list
of documents is restricted to such which have no subpath, if the root
path is selected. However, this can cause a problem if no documents on
the root path exist but only on paths below that root path.

automatically pause the crawler if there is a problem with solr

new submenu template

- added 'deleteold' option to crawler which causes that documents are
deleted which are selected by a crawl filter (host or subpath)
- site crawl used this option be default now
- made option to deleteDomain() concurrency

Fix Metadata handling
- language default on missing lang property to "uk" (fix set to nothing)
-  language set to TLD (added call to existing language calculation from TLD)
-  coordinate number exception on possible lat/lon content of "NaN,NaN"

adjust Netbeans IDE classpath (for Solr/Lucene 4.0.0 jars)

host browser now shows also number of pending files per subdirectory +
bugfixes

code cleanup

update to libraries required by solr 4.0.0

- fixed the delete option in host browser
- added a delete method which can be used to delete a full subpath in
solr.

added the MIME attribute for the R tag in GSA search result writer

added the host browser as link to search results. that means you can
select a browsing position after a search is done on the search results.

more refactoring - integrated the code of SnippetProcess into
SearchEvent

missing license-files
(sorry I didn't commit theses files by mistake)

added missing libraries

bump to httpclient-4.2.2

added more / all new crawl profile fields into crawl profile editor

in case that a crawl profile has a collection assigned, use the
collection to show a name in the web interface. This should prevent that
much too long names make the interface unusable.

enhaced data structures for balancer and latency computation which
should produce a bit better prognosis about forced waiting times.

removed options for stopwords which are not used

added the Google Search Appliance (GSA) api interface to the main menu.
See:
https://developers.google.com/search-appliance/documentation/68/xml_reference#request_overview

less latency

better balancing and duetime-cumputation also for no-delay intranet
hosts

disabled writing new entries to crawl stacks to prevent that a domain
with many documents block refreshing of the crawl queue

- fix for number of words log message
- adding meta:refresh also to crawler stack

- added concurrency for robots.txt loading
- changed data model for domain counter

enhancement to solr caching: consider that during a get() the document
is not in solr but the cache points out that a commit is needed to get
the document.

more auto-commit calls when a search interface is opened, but not when a
search is done there to prevent blocking during search-time.

if a network configuration is choosed which does not allow DHT and no
P2P communication is in robinson mode) then some menu entries are
disabled which have no use in this mode.

enhanced solr caching:
- increased cache size which is needed for longer solr commit time
- speed hacks on cache write code

moved static method from ClusteredScoreMap to MapDataMining because it
was not used in the ClusteredScoreMap class but only in MapDataMining

- optimize code of augmented parsing to enhence document tags
- commented out augmentedparser.analyse (not function implemented yet)
- adjust init of document title list to always use same list type

force a commit in advance of a search for the administrator to get most
recent results even if commit time is high and an indexing is ongoing.

added an option to force a commit to solr.
may be used by a search front-end in case that the commitWithinMs time
is too short to get recently indexed documents.

rise commitWithinMs to default-value from SwitchBoard
(result in lower hd-io)

no dots in memory-graph (there are to much of them)

another performance and memory hack to graphics: this makes it possible
to produce a 100-Megapixel png network graphic image on my 6 year old
laptop in standard configuration in 10 seconds.

- show more lines in online log
- reverse order is default now

more image processing hacks

because the new PngEncoder had a problem with the PixelGrabber which is
caused by a JRE bug, the PixelGrabber had to be circumvented using an
own frame buffer which can be read without a PixelGrabber. This resulted
in ultra-fast and much less memory-consuming transformation. YaCy images
are now generated really fast!

when a new crawl is started, an equal crawl, if still running, is
terminated and the corresponding crawl profile is deleted (this also
clears the crawl queue entries for that crawl profile)

the web structure image shows the pivot dot in a different color

- prepared PngEncoder for concurrency: PixelGrabber.grabPixels is the
main time-consuming process. This shall be done in concurrency.
- added concurrent processes to call the PixelGrabber and framework to
do that (queues)

It is now possible to create 4k-Images (3840x2160) i.e. with the Network
Graphics servlet

- new order of data computation: first compute the size of
compressed deflater output, then assign an exact-sized byte[] which
makes resizing afterwards superfluous
- after all enhancements all class objects were removed; result is just
one short static method
- made objects final where possible

added a 9-year old png encoder from David Eisenberg which I rewrote
quite a bit to remove all code that handles transparency. With this
highly specialized png writer it is possible to write png images much
faster that with the JRE built-in png writer.
In a second step it can be possible to add concurrency to increase
computation speed further.

added option to view the complete directory structure in host browser

enhanced web structure images

gsa results shall have only one title in metadata and that should be the
visible title in the <title>-tag

whitelist yacyportalsearch aka search.yacy.net

showing the web structure graph as animation in the crawl monitor

- fix: with augmented parsing = on; missing metadata in index (like title) due to overwriting metadata by adding multiple result docs from augmentparser with same url
- fix Document.addsubdocuments: sections might be initialized as Arrays.toList which does not provide the used .addAll methode
   see e.g. http://kamleshkr.wordpress.com/2010/02/17/inside-java-arrays-aslistt-a/

enhanced webstructure image: introduced
- multiple hosts can be listed (comma-separated) as host argument
- new 'bf'-attribut (branch factor): the maximum number of edges per
node
- the bf-value is computed automatically
- ordering of nodes when the graphic is drawed: mostly the drawing ends
with an limitation eg. number of nodes. When this happens, it should be
ensured that more 'interesting' nodes are painted in advance. This is
now done by sorting all nodes by the number of links they have in de
distant sub-graph.

smaller dhtDispatcher.cloudSize
@Orbiter: we talked about this times ago - please revert if I'm wrong

not hold a expensive cache of references for DHT-out,but but load them
on demand
see: http://forum.yacy-websuche.de/viewtopic.php?f=8&t=4530

format crawler timeout output string in seconds (was days)

more custom field usage in gsa search result

- more refactoring / private methods
- fix for usage of custom solr field names

added a HostBrowser.xml api file and changed a bit of attribute naming

added a shell script which can be used to delete the api action steering
table. This may be necessary if the api is called by remote command and
the recordings are not used. Then they can be deleted frequently by
calling this clear command using a cron job

added a shell script which can be used to add a rss feed to the index.
All pages linked in the rss feed are added. The process is not repeated
automatically. If you want to repeat this, add the command to a cron
job.

specified more URIMetadata as URIMetadataNode

added date info in parser errors

use less cache

default cache size was much too high; decreased solr cache size

enhancement to post argument parsing - possible fix to zero-filled
parameter values

less solr prefetch

added a crawl start checker which makes a simple analysis on the list of
all given urls: shows if the url can be loaded and if there is a robots
and/or a sitemap.

moved the index deletion functions from IndexControlRWIs to
IndexControlURLs where it appears more naturally. Because the RWI
administration is less important in the presence of Solr, the
IndexControlURL is now the default servlet when the Index Administration
button on the main menu is selected.

- add language detection from <html lang="xx"> tag
- add jaudiotagger jar to Netbeans-IDE project classpath

added Open Graph Metadata default fields, see http://ogp.me/ns#

added schema.org breadcrumb counter to parser and solr schema

replaced some more .getBytes() with UTF8/ASCII.getBytes()

added an url rewriter which can be used to remove session ids from urls

use links in AccessTracker

enhanced the host browser

adjusted Netbeans-IDE classpath to current jars
   change solr jars to 3.6.1 (from 3.6.0)
   change lucene jars to 3.6.1 (from 3.6.0)
   added jsoup-1.6.3

- add translation for ConfigHeuristics_p.html # section search-result 
- removed old/unused scroogle text

bump to httpcore-4.2.2 (maintenance release)

refactoring

added an option to start indexing right from the host browser

added the usage of synonyms to the GSA search interface

turned the synonyms_t Text field into a multi-valued String field
synonyms_sxt

ups, added missing class for last commit

added an underline text field to solr to record all underlined texts

The HostBrowser now offers to index files that are discovered because
they are linked in the web interface.

fixed computation of links in host browser which are not indexed but
knwon by the crawler. Such links are now displayed in grey color.

added nice links to the host browser:
- click on the file icon to get the metadata of the file
- click on the link icon behind the link to open the original file in
the browser

added lucene memory library which is now necessary as solr has to
process more complex queries

another fix for the debian installer: the installer fails because some
classes had unresolved dependencies. This fix removes the dependencies.

allow Cross-Origin Resource Sharing for all stream servlets, that is the
solr and the gsa search interface. That means that all JavaScript in
browsers now can Cross-Origin access all YaCy search interfaces, which
opens the option of 'YaCy Client in Browser' and 'End-Point Fail-over'
concepts.

fixed url search in IndexControlURLs_p.html / using now the solr
interface

increased strength of crawling waves in network image

force usage of default faceting mechanisms for search

- better date ranking
- more protection against NPE and time travel effects

- if a "/date" modifier is used, the solr remote query applies an
ordering by date (ascending)
- added also some 'anti-timetravel' protection (check if date is in the
future within any metadata date field)

We assert that no other metadata storage than solr is used now.
Therefore a property like solrConnected() must be true all the time.
Removal of this method causes removal of all write operations to the old
metadata index.

made the index schema retrieval public and allow cross-domain retrieval

enhanced snippet extractor to find snippets also inside of tokens of an
url

added filename for missing crawlname when crawling from file

pdf- and zipParser should not use forced Memory-Limits

adding CORS access header for Network.xml to overcome cross domain
restriction (e.g. necessary to build a JavaScript YaCy
client).

better abstraction for solr query params

- fix for NPEs during remote solr configuration
- fixed remote solr setting switch
- added more logging

added dummy update servlet

removed tenant query attribute since it is not used any more and is
replaced by the site-operator in the GSA interface. This operator can
also be simulated in the Solr interface using the collections_sxt field.

using the search filter to drill down search to file types.
A search like "mp3 filetype:mp3" will now maybe surprise you.

more cleaning (yacy-cora)

added the indexrestore.sh script which must be called with the path of
the index dump. This is the reverse of indexdump.sh which takes the
output of indexdump.sh as input to restore an index.
Now it should be possible to transfer a complete YaCy Solr index from
one peer yacy1 to another peer yacy2 with the following command:
yacy2/bin/indexrestore.sh ´yacy1/bin/indexdump.sh´

- added xml output in IndexControlURLs to get the storage page of index
dump commands
- adjusted the apicall.sh script to get the downloaded text as output to
stdout which is necessary to parse the content out of it
- added indexdump.sh script which creates a solr dump and prints out the
storage path for the index dump
- added synchronization to the Fulltext class to prevent that data is
stored to a non-existing solr index while this index is disabled during
the storage of the dump

used the new zip writer/reader to add a solr dump process: the whole
solr index can be written to a zip dump and also restored during runtime

added a directory-to-zip writer and zip-to-directory reader

a bit more logging

simplifications in DHT Distribution class and more documentation

simplified DHT classes

added new classes to renovate the YaCy protocol based on simple data
structures in cora:
- added the Peer object, which is a fresh version of Seed
- added the Peers object, which is a fresh version of Network
- added the Network api access class to retrieve a list of peers based
on the Network.xml servlet in all YaCy peers.

fixed mistake in wt-option which caused that the yacy json format
overlapped the solr built-in json format

added HostBrowser servlet (stub)

more ignore

added ftp to getName

change parameter to support the smw extension for list import

full memory usage for debian and when changing the size: debian seems to
dislike the big difference between xmx and xms (I have crashes here
which stop if both values are same)

added new crawl options:
- indexUrlMustMatch and indexUrlMustNotMatch which can be used to select
loaded pages for indexing. Default patterns are in such a way that all
loaded pages are also indexed (as before) but when doing an expert crawl
start, then the user may select only specific urls to be indexed.
- crawlerNoDepthLimitMatch is a new pattern that can be used to remove
the crawl depth limitation. This filter a never-match by default (which
causes that the depth is used) but the user can select paths which will
be loaded completely even if a crawl depth is reached.

fixed the size() method which counted also failed pages (which are also
inside the solr index)

added new crawl attributes in crawl profile (not active yet)

added default facet fields for json response format (stub)

added missing license headers

added a regular expression test servlet which is linked within the
parser/crawler error page whenever a problem with regular expression
occurs.
This makes it easy to correct and enhance the must-match and
must-not-match patterns just by trying out which pattern could be
correct.

added twitter search heuristic

- some corrections in usage of getFile() and getFileName()
- added more attributes in json response writer according to yacy
servlet

added the protocol and the file name extension to the solr fields since
these fields are probably facets in file search

no complaints about memory if the database is empty

activate two solr fields which will be used by administration interface
(later)

added facet stub in JsonResponseWriter

enhanced solr writers

added search functionality to ViewFile.html servlet

- added collections to yacydoc
- changed yacydoc.htm to yacydoc.json
- added query logging in solr and gsa search result

- added a json writer for solr (yes there was one using xslt but this
one writes the same way as yacysearch.json)
- using the new json solr result to change the ajax search in
IndexControlURLs to the new solr search

Merge remote-tracking branch 'reger/master'

removed warnings

Added more sophisticated RDF output for YMarks, including the folder
structure (b:Topic) and support for multiple tags (dc:subject) and
folders (b:hasTopic) via rdf:Bag container.

keep input order of translation entries within one file section.
Allowing on translation conflicts (translaton of words contained in other sentence) to put shorter key at the end of the translation list.

added Solr fields:
inboundlinks_text_chars_val
inboundlinks_text_words_val
inboundlinks_alttag_txt
outboundlinks_text_chars_val
outboundlinks_text_words_val
outboundlinks_alttag_txt

added solr field images_withalt_i

added disjunction '|' option to site parameter in GSA API

clear fulltext-cache and stop crawling if running out of memory

also do a clearcache on the solr-connector-caches

statistics for solr-cache

added collections to crawl monitor

added h1..h6 counter fields

add nice submit buttons to pdblue skin

dependency is java6 only

removed jquery.slider as it is already included as part of jquery-ui
package

removed unused jquery plugin slider as it is part of jquery-ui package

added synchronization to solr server requests since lucene is not
thread-safe. We experienced problems as described in
http://stackoverflow.com/questions/5327978/lockobtainfailedexception-updating-lucene-search-index-using-solr

removed many warnings

- moved the gsa search interface from /gsa/searchresult? to /gsa/search?
- fixed the NB field data

fixed problems with GSA api:
- better FS attribute
- highlightning of searched words in title

- fixed num parameter in GSA api
- changed FS attribute in GSA api

added new field for solr:
url_paths_sxt
url_parameter_i
url_parameter_key_sxt
url_parameter_value_sxt
url_chars_i

content control: apply filter if enabled to crawls

Merge commit '65d49df865f60511d22d86fb15c33a082176e7ab'

added boosts to solr search queries

switched off some solr logging

added gsa result attribute 'has'

security fix: clear automtic password only if adminAccountForLocalhost=false to prevent remote access to protected pages after restart.
if adminAccountForLocalhost=true leave automatic password unchanged so access from local host is granted but remote access is preventet from the 1st second.

- correct length computation for BStringObject (bugfix suggested by
apfelmaennchen)
- using ASCII for string conversion for Strings generated from Integer 

- added hack to prevent that stream servlet paths are not parsed wrongly
if the path contains a dot.
- added also warnings if documents are requests which do not exist.