YaCy Release 1.6

Main Release 1.6

Major Changes   
Jump to: Bugfixes / Other Changes

CommitDescription
Sat Jul 27 10:19:46 CEST 2013
by Roland Haeder
Fix for NPE:
E 2013/07/26 20:29:29 BUSYTHREAD Runtime Error in
serverInstantThread.job, thread
'net.yacy.search.Switchboard.cleanupJob': null; target exception: null
java.lang.NullPointerException
        at
net.yacy.search.schema.CollectionConfiguration.convergenceStep(CollectionConfiguration.java:1116)
        at
net.yacy.search.schema.CollectionConfiguration.postprocessing(CollectionConfiguration.java:897)
        at net.yacy.search.Switchboard.cleanupJob(Switchboard.java:2296)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:107)
        at
net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:165)

Conflicts:
	source/net/yacy/search/schema/CollectionConfiguration.java
Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java
Thu Jul 25 12:17:29 CEST 2013
by Michael Peter Christen
always use a default value for boolean options to have transparency for
the outcome if the attribute is missing in servlets
Changed Files: htroot/AccessPicture_p.java, htroot/ConfigPortal.java, htroot/ConfigSearchPage_p.java, htroot/ConfigUpdate_p.java, htroot/Connections_p.java, htroot/Crawler_p.java, htroot/HostBrowser.java, htroot/IndexFederated_p.java, htroot/NetworkPicture.java, htroot/PeerLoadPicture.java, htroot/Status.java, htroot/Table_API_p.java, htroot/Threaddump_p.java, htroot/ViewFile.java, htroot/api/ymarks/import_ymark.java, htroot/opensearchdescription.java, source/net/yacy/server/serverObjects.java
Wed Jul 17 18:37:34 CEST 2013
by Roland Haeder
Fixes and avoid of catching bad exceptions (some):
- Rewrote usage of HashMap/Map to concurrent versions (to avoid a
CME=ConcurrentModificationException)
- Rewrote ConnectionInfo (as an example) to use a synchronized iterator
instead of synchronizing an
  already synced HashSet (see Collections call)
- This avoids catching CMEs again
- Commented out noisy ConcurrentLog.logException() call

Conflicts:
	source/net/yacy/repository/LoaderDispatcher.java
Changed Files: htroot/AccessPicture_p.java, htroot/AccessTracker_p.java, htroot/CrawlStartScanner_p.java, htroot/PerformanceMemory_p.java, htroot/ServerScannerList.java, htroot/suggest.java, htroot/yacy/hello.java, htroot/yacysearch.java, source/net/yacy/cora/document/RSSFeed.java, source/net/yacy/cora/document/WordCache.java, source/net/yacy/cora/protocol/ConnectionInfo.java, source/net/yacy/peers/DHTSelection.java, source/net/yacy/repository/LoaderDispatcher.java
Wed Jul 17 18:31:30 CEST 2013
by Roland Haeder
Added 'final' for all exception blocks as this helps the Java compiler
to optimize memory usage

Conflicts:
	source/net/yacy/search/Switchboard.java
Changed Files: htroot/AccessTracker_p.java, htroot/Bookmarks.java, htroot/CacheResource_p.java, htroot/ConfigHeuristics_p.java, htroot/ContentIntegrationPHPBB3_p.java, htroot/CrawlCheck_p.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/HostBrowser.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexDeletion_p.java, htroot/IndexFederated_p.java, htroot/IndexSchema_p.java, htroot/RankingSolr_p.java, htroot/RegexTest.java, htroot/ServerScannerList.java, htroot/Table_API_p.java, htroot/Table_YMark_p.java, htroot/Threaddump_p.java, htroot/Triple_p.java, htroot/Vocabulary_p.java, htroot/YMarks.java, htroot/api/bookmarks/get_bookmarks.java, htroot/api/bookmarks/get_folders.java, htroot/api/bookmarks/posts/all.java, htroot/api/bookmarks/posts/get.java, htroot/api/bookmarks/tags/addTag_p.java, htroot/api/bookmarks/xbel/xbel.java, htroot/api/citation.java, htroot/api/timeline.java, htroot/api/webstructure.java, htroot/api/ymarks/import_ymark.java, htroot/api/ynetSearch.java, htroot/gsa/searchresult.java, htroot/interaction/GetRDF.java, htroot/processing/domaingraph/applet/domaingraph.java, htroot/solr/select.java, htroot/suggest.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacysearch.java, htroot/yacysearchitem.java, libbuild/GitRevTask/GitRevTask.java, libbuild/svnRevNr/svnRevNrParser.java, source/net/yacy/YaCySearchClient.java, source/net/yacy/contentcontrol/ContentControlFilterUpdateThread.java, source/net/yacy/contentcontrol/SMWListImporter.java, source/net/yacy/contentcontrol/SMWListImporterFormatObsolete.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/ai/example/testorder.java, source/net/yacy/cora/ai/greedy/Agent.java, source/net/yacy/cora/ai/greedy/Attempts.java, source/net/yacy/cora/ai/greedy/Context.java, source/net/yacy/cora/ai/greedy/Engine.java, source/net/yacy/cora/document/JSONArray.java, source/net/yacy/cora/document/JSONObject.java, source/net/yacy/cora/document/JSONTokener.java, source/net/yacy/cora/document/RSSFeed.java, source/net/yacy/cora/document/RSSReader.java, source/net/yacy/cora/federate/opensearch/OpenSearchConnector.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ShardSelection.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/federate/solr/instance/ServerShard.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/cora/federate/yacy/Peer.java, source/net/yacy/cora/federate/yacy/Peers.java, source/net/yacy/cora/federate/yacy/api/Network.java, source/net/yacy/cora/geo/GeonamesLocation.java, source/net/yacy/cora/geo/OpenGeoDBLocation.java, source/net/yacy/cora/language/phonetic/Phonetic.java, source/net/yacy/cora/language/synonyms/AutotaggingLibrary.java, source/net/yacy/cora/language/synonyms/SynonymLibrary.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/order/Digest.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/LinkExtractor.java, source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java, source/net/yacy/cora/storage/ConcurrentARC.java, source/net/yacy/cora/storage/Configuration.java, source/net/yacy/cora/storage/Files.java, source/net/yacy/cora/storage/SizeLimitedSet.java, source/net/yacy/cora/storage/ZIPWriter.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/Request.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/crawler/retrieval/URLRewriterLibrary.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/data/BlogBoard.java, source/net/yacy/data/BookmarkDate.java, source/net/yacy/data/BookmarksDB.java, source/net/yacy/data/DidYouMean.java, source/net/yacy/data/MessageBoard.java, source/net/yacy/data/UserDB.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/wiki/WikiBoard.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkDMOZImporter.java, source/net/yacy/data/ymark/YMarkEntry.java, source/net/yacy/data/ymark/YMarkHTMLImporter.java, source/net/yacy/data/ymark/YMarkImporter.java, source/net/yacy/data/ymark/YMarkJSONImporter.java, source/net/yacy/data/ymark/YMarkRDF.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/data/ymark/YMarkXBELImporter.java, source/net/yacy/document/Document.java, source/net/yacy/document/ImageParser.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/SurrogateReader.java, source/net/yacy/document/content/dao/DatabaseConnection.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/html/CharacterCoding.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaTripleImpl.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xml/ODContentHandler.java, source/net/yacy/document/parser/xml/opensearchdescriptionReader.java, source/net/yacy/gui/framework/Application.java, source/net/yacy/gui/framework/Browser.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BEncodedHeapBag.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/Gap.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapColumnIndex.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Stack.java, source/net/yacy/kelondro/blob/Stacks.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/blob/TablesColumnBLOBIndex.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/io/ByteCountInputStream.java, source/net/yacy/kelondro/io/CachedFileReader.java, source/net/yacy/kelondro/io/CachedFileWriter.java, source/net/yacy/kelondro/io/Records.java, source/net/yacy/kelondro/logging/LogalizerHandler.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/BDecoder.java, source/net/yacy/kelondro/util/BEncoder.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/ReverseMapIterator.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/migration.java, source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/graphics/EncodedImage.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/ResourceObserver.java, source/net/yacy/search/Shutdown.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/ReindexSolrBusyThread.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SecondarySearchSuperviser.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/server/http/ChunkedInputStream.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/serverObjects.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/upnp/Discovery.java, source/net/yacy/upnp/DiscoveryAdvertisement.java, source/net/yacy/upnp/DiscoveryListener.java, source/net/yacy/upnp/JXPathParser.java, source/net/yacy/upnp/ServicesEventing.java, source/net/yacy/upnp/devices/UPNPRootDevice.java, source/net/yacy/upnp/impls/InternetGatewayDevice.java, source/net/yacy/upnp/messages/ActionMessage.java, source/net/yacy/upnp/messages/ActionMessageResponseParser.java, source/net/yacy/upnp/messages/StateVariableMessage.java, source/net/yacy/upnp/messages/StateVariableResponseParser.java, source/net/yacy/upnp/services/ISO8601Date.java, source/net/yacy/upnp/services/UPNPService.java, source/net/yacy/utils/CryptoLib.java, source/net/yacy/utils/SignatureOutputStream.java, source/net/yacy/utils/UPnP.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/utils/tarTools.java, source/net/yacy/visualization/AnimationGIF.java, source/net/yacy/visualization/RasterPlotter.java, source/net/yacy/yacy.java, test/de/anomic/document/ParserTest.java, test/de/anomic/yacy/yacyURLTest.java, test/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnectorTest.java
Wed Jul 17 15:20:56 CEST 2013
by Michael Peter Christen
added collection_sxt to error documents
Changed Files: htroot/Crawler_p.java, htroot/yacy/crawlReceipt.java, htroot/yacy/urls.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/snippet/MediaSnippet.java
Sat Jul 13 13:12:36 CEST 2013
by Roland Haeder
Fixed CHMOD on many files + added missing loggers (e.g. jena) and made some noisy loggers quiet
Changed Files: addon/YaCy.app/Contents/MacOS/JavaApplicationStub, addon/testkeys, defaults/solr/schema.xml, defaults/solr/solrconfig.xml, defaults/yacy.logging, htroot/api/feed.rss, htroot/env/terminal.css, htroot/js/highslide/graphics/controlbar2.gif, htroot/js/highslide/graphics/controlbar3.gif, htroot/js/highslide/graphics/controlbar4-hover.gif, htroot/js/highslide/graphics/controlbar4.gif, htroot/js/highslide/graphics/fullexpand.gif, htroot/js/highslide/graphics/geckodimmer.png, htroot/js/highslide/graphics/loader.gif, htroot/js/highslide/graphics/loader.white.gif, htroot/js/highslide/graphics/outlines/Outlines.psd, htroot/js/highslide/graphics/outlines/beveled.png, htroot/js/highslide/graphics/outlines/drop-shadow.png, htroot/js/highslide/graphics/outlines/glossy-dark.png, htroot/js/highslide/graphics/outlines/outer-glow.png, htroot/js/highslide/graphics/outlines/rounded-black.png, htroot/js/highslide/graphics/outlines/rounded-white.png, htroot/js/highslide/graphics/resize.gif, htroot/js/highslide/graphics/zoomin.cur, htroot/js/highslide/graphics/zoomout.cur, htroot/js/highslide/highslide.js, htroot/js/query.js, htroot/js/rss2.js, htroot/processing/domaingraph/applet/domaingraph.jar, htroot/processing/domaingraph/applet/domaingraph.pde, htroot/processing/domaingraph/applet/loading.gif, htroot/processing/domaingraph/domaingraph.pde, htroot/processing/put_in_libraries/animation.zip, htroot/processing/put_in_libraries/physics.zip, htroot/yacy/ui/colorpicker_images/Thumbs.db, htroot/yacy/ui/colorpicker_images/blank.gif, htroot/yacy/ui/colorpicker_images/colorpicker_background.png, htroot/yacy/ui/colorpicker_images/colorpicker_hex.png, htroot/yacy/ui/colorpicker_images/colorpicker_hsb_b.png, htroot/yacy/ui/colorpicker_images/colorpicker_hsb_h.png, htroot/yacy/ui/colorpicker_images/colorpicker_hsb_s.png, htroot/yacy/ui/colorpicker_images/colorpicker_indic.gif, htroot/yacy/ui/colorpicker_images/colorpicker_overlay.png, htroot/yacy/ui/colorpicker_images/colorpicker_rgb_b.png, htroot/yacy/ui/colorpicker_images/colorpicker_rgb_g.png, htroot/yacy/ui/colorpicker_images/colorpicker_rgb_r.png, htroot/yacy/ui/colorpicker_images/colorpicker_select.gif, htroot/yacy/ui/colorpicker_images/colorpicker_submit.png, htroot/yacy/ui/colorpicker_images/custom_background.png, htroot/yacy/ui/colorpicker_images/custom_hex.png, htroot/yacy/ui/colorpicker_images/custom_hsb_b.png, htroot/yacy/ui/colorpicker_images/custom_hsb_h.png, htroot/yacy/ui/colorpicker_images/custom_hsb_s.png, htroot/yacy/ui/colorpicker_images/custom_indic.gif, htroot/yacy/ui/colorpicker_images/custom_rgb_b.png, htroot/yacy/ui/colorpicker_images/custom_rgb_g.png, htroot/yacy/ui/colorpicker_images/custom_rgb_r.png, htroot/yacy/ui/colorpicker_images/custom_submit.png, htroot/yacy/ui/colorpicker_images/select.png, htroot/yacy/ui/colorpicker_images/select2.png, htroot/yacy/ui/colorpicker_images/slider.png, htroot/yacy/ui/css/colorpicker.css, htroot/yacy/ui/js/jquery-1.3.2.min.js, htroot/yacy/ui/js/jquery-ui-1.7.2.min.js, htroot/yacy/ui/js/jquery.tagcloud.min.js, htroot/yacy/ui/js/jquery.tinysort.min.js, lib/jcifs-1.3.15-LICENSE.txt
Fri Jul 12 16:24:56 CEST 2013
by Michael Peter Christen
refactoring of load_delay: this is a matter of client identification
Changed Files: htroot/CrawlCheck_p.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/Load_RSS_p.java, htroot/ViewFile.java, htroot/ViewImage.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/webstructure.java, source/net/yacy/cora/protocol/ClientIdentification.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java
Tue Jul 09 14:42:39 CEST 2013
by Michael Peter Christen
replaced usage of log4j by ConcurrentLog where possible
Changed Files: source/net/yacy/cora/document/WordCache.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/yacy/Peers.java, source/net/yacy/cora/geo/GeonamesLocation.java, source/net/yacy/cora/language/synonyms/AutotaggingLibrary.java, source/net/yacy/cora/language/synonyms/SynonymLibrary.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/TimeoutRequest.java, source/net/yacy/crawler/retrieval/URLRewriterLibrary.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/kelondro/logging/SimpleLogFormatter.java
Tue Jul 09 14:28:25 CEST 2013
by Michael Peter Christen
- refactoring of log to ConcurrentLog:
jdk-based logger tend to block
at java.util.logging.Logger.log(Logger.java:476) in concurrent
environments. This makes logging a main performance issue. To overcome
this problem, this is a add-on to jdk logging to put log entries on a
concurrent message queue and log the messages one by one using a
separate process.
- FTPClient uses the concurrent logging instead of the log4j logger
Changed Files: htroot/BlacklistCleaner_p.java, htroot/Blacklist_p.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/CacheResource_p.java, htroot/ConfigAccounts_p.java, htroot/ConfigHeuristics_p.java, htroot/ConfigPortal.java, htroot/ConfigSearchPage_p.java, htroot/ConfigUpdate_p.java, htroot/Connections_p.java, htroot/ContentIntegrationPHPBB3_p.java, htroot/CrawlCheck_p.java, htroot/CrawlProfileEditor_p.java, htroot/CrawlResults.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/HostBrowser.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexCreateQueues_p.java, htroot/IndexFederated_p.java, htroot/IndexImportOAIPMH_p.java, htroot/IndexSchema_p.java, htroot/Load_RSS_p.java, htroot/NetworkPicture.java, htroot/News.java, htroot/ProxyIndexingMonitor_p.java, htroot/ServerScannerList.java, htroot/Steering.java, htroot/Table_API_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.java, htroot/Triple_p.java, htroot/User.java, htroot/ViewImage.java, htroot/ViewLog_p.java, htroot/ViewProfile.java, htroot/Vocabulary_p.java, htroot/WebStructurePicture_p.java, htroot/YMarks.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/table_p.java, htroot/api/termlist_p.java, htroot/api/timeline.java, htroot/api/webstructure.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/delete_ymark.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/get_xbel.java, htroot/api/ymarks/get_ymark.java, htroot/api/ymarks/import_ymark.java, htroot/api/ymarks/manage_tags.java, htroot/gsa/searchresult.java, htroot/rct_p.java, htroot/solr/select.java, htroot/solr/update.java, htroot/yacy/crawlReceipt.java, htroot/yacy/hello.java, htroot/yacy/message.java, htroot/yacy/query.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/contentcontrol/SMWListImporter.java, source/net/yacy/contentcontrol/SMWListImporterFormatObsolete.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/federate/opensearch/OpenSearchConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ShardSelection.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/geo/OpenGeoDBLocation.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/util/ConcurrentLog.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/NoticedURL.java, source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/crawler/retrieval/SitemapImporter.java, source/net/yacy/data/BlogBoard.java, source/net/yacy/data/BlogBoardComments.java, source/net/yacy/data/BookmarkDate.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/data/BookmarksDB.java, source/net/yacy/data/DidYouMean.java, source/net/yacy/data/MessageBoard.java, source/net/yacy/data/Translator.java, source/net/yacy/data/UserDB.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/wiki/WikiBoard.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkEntry.java, source/net/yacy/data/ymark/YMarkHTMLImporter.java, source/net/yacy/data/ymark/YMarkImporter.java, source/net/yacy/data/ymark/YMarkJSONImporter.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/data/ymark/YMarkXBELImporter.java, source/net/yacy/dbtest.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/SurrogateReader.java, source/net/yacy/document/content/dao/DatabaseConnection.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/language/LanguageStatistics.java, source/net/yacy/document/language/LanguageStatisticsHolder.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/Evaluation.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/images/bmpParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/images/icoParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaTripleImpl.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/document/parser/xml/ODContentHandler.java, source/net/yacy/document/parser/xml/opensearchdescriptionReader.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/gui/framework/Browser.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BEncodedHeapBag.java, source/net/yacy/kelondro/blob/BEncodedHeapShard.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Stack.java, source/net/yacy/kelondro/blob/Stacks.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/blob/TablesColumnBLOBIndex.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/RAMIndexCluster.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/ByteCountInputStream.java, source/net/yacy/kelondro/io/CachedFileReader.java, source/net/yacy/kelondro/io/CachedFileWriter.java, source/net/yacy/kelondro/io/CachedRecords.java, source/net/yacy/kelondro/io/Records.java, source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java, source/net/yacy/kelondro/logging/SimpleLogFormatter.java, source/net/yacy/kelondro/rwi/AbstractIndex.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/table/ChunkIterator.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SQLTable.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/ConsoleInterface.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/GenerationMemoryStrategy.java, source/net/yacy/kelondro/util/MemoryStrategy.java, source/net/yacy/kelondro/util/MergeIterator.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/kelondro/util/StandardMemoryStrategy.java, source/net/yacy/kelondro/workflow/AbstractBlockingThread.java, source/net/yacy/kelondro/workflow/AbstractBusyThread.java, source/net/yacy/kelondro/workflow/AbstractThread.java, source/net/yacy/kelondro/workflow/InstantBlockingThread.java, source/net/yacy/kelondro/workflow/InstantBusyThread.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/migration.java, source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/Transmission.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/peers/operation/yacyVersion.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/MemoryTracker.java, source/net/yacy/search/ResourceObserver.java, source/net/yacy/search/Shutdown.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/ReindexSolrBusyThread.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SearchEventCache.java, source/net/yacy/search/query/SecondarySearchSuperviser.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/RankingProfile.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/http/TemplateEngine.java, source/net/yacy/server/serverCore.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/server/serverSwitchAbstractAction.java, source/net/yacy/utils/UPnP.java, source/net/yacy/utils/crypt.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/utils/gzip.java, source/net/yacy/utils/tarTools.java, source/net/yacy/visualization/Captcha.java, source/net/yacy/visualization/ChartPlotter.java, source/net/yacy/visualization/RasterPlotter.java, source/net/yacy/yacy.java
Wed Jul 03 14:50:06 CEST 2013
by Michael Peter Christen
added a crawl option to obey html-meta-robots-noindex. This is on by
default.
Changed Files: defaults/yacy.init, htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java, htroot/CrawlStartSite_p.html, htroot/Crawler_p.java, htroot/QuickCrawlLink_p.html, htroot/QuickCrawlLink_p.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionConfiguration.java
Tue Jun 25 16:27:20 CEST 2013
by Michael Peter Christen
added url_file_name_s in default collection schema for the file name
without the file extension. This part of the file path is removed from
the multi-field url_paths_sxt, which has now not the file name as last
part of the path list.

The same applies to the new fields source_file_name_s and
target_file_name_s in the webgraph schema.
Changed Files: defaults/solr.collection.schema, defaults/solr.webgraph.schema, htroot/ViewFile.java, htroot/yacysearchitem.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/document/analysis/Classification.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/responsewriter/JsonResponseWriter.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/crawler/data/ResultImages.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java


Bugfixes   
Jump to: YaCy Release 1.6 top / Other Changes

CommitDescription
Thu Jul 25 09:53:19 CEST 2013
by Michael Peter Christen
fix in forced deletion: forced commit needed
Changed Files: source/net/yacy/search/Switchboard.java
Thu Jul 25 09:31:48 CEST 2013
by Michael Peter Christen
new test case for http://bugs.yacy.net/view.php?id=141
Changed Files: source/net/yacy/cora/document/MultiProtocolURI.java
Thu Jul 25 09:26:24 CEST 2013
by Michael Peter Christen
fix for http://bugs.yacy.net/view.php?id=267
Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java
Thu Jul 25 09:21:37 CEST 2013
by Michael Peter Christen
fix for http://bugs.yacy.net/view.php?id=268
Changed Files: source/net/yacy/search/index/Segment.java
Tue Jul 23 16:46:44 CEST 2013
by orbiter
fixes for problems with remote solr and non-activated webgraph index
Changed Files: htroot/IndexDeletion_p.java, htroot/IndexFederated_p.java, source/net/yacy/cora/federate/opensearch/OpenSearchConnector.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/WebgraphConfiguration.java
Mon Jul 22 15:55:20 CEST 2013
by Michael Peter Christen
fixed problem with remote luke requests
Changed Files: source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/solr/instance/ResponseAccumulator.java
Thu Jul 18 12:37:32 CEST 2013
by Michael Peter Christen
fix for NPE & bug #264
Changed Files: source/net/yacy/cora/federate/solr/instance/InstanceMirror.java
Wed Jul 17 12:41:54 CEST 2013
by Michael Peter Christen
fixes using the embedded connector
Changed Files: htroot/IndexReIndexMonitor_p.java, source/net/yacy/search/index/ReindexSolrBusyThread.java
Tue Jul 16 18:18:55 CEST 2013
by Michael Peter Christen
fix for crawl stack domain counter
Changed Files: source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/NoticedURL.java, source/net/yacy/search/Switchboard.java
Sun Jul 14 17:40:28 CEST 2013
by orbiter
fix for bug 260
Changed Files: source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/search/index/ReindexSolrBusyThread.java
Sat Jul 13 11:10:05 CEST 2013
by orbiter
npe fix
Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java
Fri Jul 12 14:38:30 CEST 2013
by orbiter
fix for http://bugs.yacy.net/view.php?id=255
Changed Files: source/net/yacy/crawler/data/CrawlQueues.java
Thu Jul 11 12:53:39 CEST 2013
by Michael Peter Christen
another patch to prevent CLOSE_WAIT status on solr connections
Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java
Thu Jul 11 12:53:16 CEST 2013
by Michael Peter Christen
fixed parsing of peer flags
Changed Files: source/net/yacy/peers/Seed.java
Thu Jul 04 21:46:46 CEST 2013
by orbiter
fix for rssTerminal coloring
Changed Files: htroot/rssTerminal.html
Sun Jun 30 03:50:07 CEST 2013
by reger
bugfix: location (lat/lon) meta data retrival (Double.NaN check)
Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java
Fri Jun 28 13:16:25 CEST 2013
by Michael Peter Christen
fix for solr requests if no query part is given (prevent npe)
Changed Files: htroot/solr/select.java, source/net/yacy/peers/Protocol.java, source/net/yacy/server/serverObjects.java
Tue Jun 25 16:36:16 CEST 2013
by Michael Peter Christen
npe fix
Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java
Fri Jun 21 15:47:12 CEST 2013
by Michael Peter Christen
fix for daterange: the to-date is inclusive
Changed Files: htroot/gsa/searchresult.java


Other Changes   
Jump to: YaCy Release 1.6 top / Bugfixes

CommitDescription
Sat Jul 27 11:26:14 CEST 2013
by orbiter
Release 1.6
Changed Files: build.properties
Sat Jul 27 11:22:40 CEST 2013
by orbiter
we should not do too much greedylearning at this time as we don't have
enough experience with it. set greedylearning.limit.doccount to a much
lower limit.
Changed Files: defaults/yacy.network.freeworld.unit, defaults/yacy.network.intranet.unit, defaults/yacy.network.metager.unit, defaults/yacy.network.webportal.unit
Sat Jul 27 11:00:35 CEST 2013
by orbiter
wait a bit longer before doing the first peer ping
Changed Files: source/net/yacy/search/Switchboard.java
Sat Jul 27 10:56:02 CEST 2013
by orbiter
re-try to fetch url after a soft commit
Changed Files: htroot/CrawlResults.java
Sat Jul 27 10:51:58 CEST 2013
by orbiter
don't be noisy when doing background tasks that may be allowed to fail
Changed Files: source/net/yacy/search/Switchboard.java
Sat Jul 27 10:42:38 CEST 2013
by orbiter
apply load limitation also to dht-in
Changed Files: defaults/yacy.init, htroot/yacy/transferRWI.java
Fri Jul 26 19:51:34 CEST 2013
by Roland Haeder
Some cleanups:
- added SKINS_PATH_DEFAULT as same as LISTS_PATH_DEFAULT was added
- Added 'final' keyword to a string
Changed Files: htroot/ConfigAppearance_p.java, source/net/yacy/migration.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/SwitchboardConstants.java
Sat Jul 27 10:12:58 CEST 2013
by Roland Haeder
Use SwitchboardConstants.LISTS_PATH_DEFAULT instead of 'DATA/LISTS'

Conflicts:
	htroot/api/blacklists_p.java
Changed Files: htroot/BlacklistCleaner_p.java, htroot/BlacklistImpExp_p.java, htroot/BlacklistTest_p.java, htroot/Blacklist_p.java, htroot/api/blacklists.java, htroot/api/blacklists_p.java, htroot/sharedBlacklist_p.java
Fri Jul 26 19:17:31 CEST 2013
by Roland Haeder
Removed another duplicate occurance of Blacklist.BLACKLIST_FILENAME_FILTER
Changed Files: htroot/sharedBlacklist_p.java
Fri Jul 26 19:09:38 CEST 2013
by Roland Haeder
Fix for blacklist export, also applied the filename filter here
Changed Files: htroot/api/blacklists_p.java
Fri Jul 26 19:09:00 CEST 2013
by Roland Haeder
Removed superfluous check for files ending '.black' as the previous commit already excluded all other files (e.g. .ser dumps), added logging in catch-all block
Changed Files: htroot/BlacklistImpExp_p.java
Fri Jul 26 19:07:42 CEST 2013
by Roland Haeder
Reuse Blacklist.BLACKLIST_FILENAME_FILTER
Changed Files: htroot/BlacklistCleaner_p.java
Fri Jul 26 18:32:04 CEST 2013
by Roland Haeder
Possible fix for ticket http://bugs.yacy.net/view.php?id=270, the filter for only including *.black must be applied
Changed Files: htroot/BlacklistImpExp_p.java
Thu Jul 25 12:02:11 CEST 2013
by Roland Haeder
Fix for NPE on shutdown:
java.lang.NullPointerException
        at net.yacy.search.Switchboard.storeDocumentIndex(Switchboard.java:2732)
        at net.yacy.search.Switchboard.access00(Switchboard.java:207)
        at net.yacy.search.Switchboard.run(Switchboard.java:3049)
Changed Files: source/net/yacy/search/Switchboard.java
Sat Jul 27 09:54:51 CEST 2013
by Roland Haeder
Log this exception better

Conflicts:
	source/net/yacy/kelondro/blob/Tables.java
Changed Files: source/net/yacy/kelondro/blob/Tables.java
Fri Jul 26 17:53:28 CEST 2013
by orbiter
do not write an empty failreason in case that there is no fail. Because
of the lazy instantiation rule this value was not actually written, but
if lazy instantiation is switched on, then this causes that all crawl
starts delete all crawl-start-hosts completely because this looks for
filled error reasons.
Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java
Tue Jul 23 20:24:13 CEST 2013
by reger
- small adjustment to make sure genericParser is tried last
  -- for some documents genericParser grabs document instead of specific available parser due to unordered pick of 1st to try parser
      (like .ps .rdf files and other)
- remove redundant file extension registration
Changed Files: source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java
Tue Jul 23 19:32:32 CEST 2013
by orbiter
do not run dht ditribution if system load is over 2.5
Changed Files: defaults/yacy.init, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java
Tue Jul 23 19:10:11 CEST 2013
by orbiter
added segment sizes, postprocessing status and cpu load to crawler
monitor
Changed Files: htroot/Crawler_p.html, htroot/Crawler_p.java, htroot/js/Crawler.js
Tue Jul 23 18:20:07 CEST 2013
by orbiter
added load info to status_p.xml
Changed Files: htroot/api/status_p.java, htroot/api/status_p.xml
Tue Jul 23 18:03:33 CEST 2013
by orbiter
- added information about segment count to status_p.xml
- also moved this information from the old index structure, which is
still in use for the RWI/DHT index to that front-end
Changed Files: htroot/api/status_p.java, htroot/api/status_p.xml, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/search/index/Segment.java
Tue Jul 23 14:21:12 CEST 2013
by sixcooler
changed optimization to a segment-size of index-size/5.000.000
+ one if not idle
+ one (and force) if postprocessing
Changed Files: source/net/yacy/search/Switchboard.java
Tue Jul 23 12:21:27 CEST 2013
by Michael Peter Christen
replaced StringBuffer with StringBuilder
Changed Files: source/net/yacy/cora/document/UTF8.java, source/net/yacy/cora/language/phonetic/DoubleMetaphone.java, source/net/yacy/cora/language/phonetic/Metaphone.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/kelondro/logging/SimpleLogFormatter.java, source/net/yacy/server/http/AugmentedHtmlStream.java, source/net/yacy/upnp/ServicesEventing.java, source/net/yacy/upnp/messages/ActionMessage.java, source/net/yacy/upnp/messages/StateVariableMessage.java
Tue Jul 23 12:16:51 CEST 2013
by Michael Peter Christen
replaced more containsKey() -> get() usages by a simple get(), followed
by a test for NULL. This should increase the application speed and
reduces the lookup time for the affected methods by 50%
Changed Files: source/net/yacy/kelondro/blob/Tables.java
Mon Jul 22 18:21:37 CEST 2013
by orbiter
delayed first peer-ping to send the first ping out after the http got
up; if the ping comes before the http is up, it cannot be recognized as
senior peer (if at all). See also: http://bugs.yacy.net/view.php?id=266
Changed Files: source/net/yacy/search/Switchboard.java
Mon Jul 22 17:45:12 CEST 2013
by orbiter
added the property federated.service.solr.indexing.timeout to yacy.init
to provide a configurable time-out for solr; see also:
http://bugs.yacy.net/view.php?id=254
Changed Files: defaults/yacy.init, htroot/IndexFederated_p.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java
Mon Jul 22 17:28:20 CEST 2013
by orbiter
removed forced non-soft commits to prevent index fragmentation
Changed Files: source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java
Mon Jul 22 14:17:38 CEST 2013
by Michael Peter Christen
removed failreason_t removal patch because that causes too much
confusion using an external solr. to clean up the index after a schema
change, use the index cleaner function from the online servlet
Changed Files: source/net/yacy/search/Switchboard.java
Sun Jul 21 22:57:21 CEST 2013
by reger
extend content type detection on file extension for .tif .tiff .htm
Changed Files: source/net/yacy/crawler/retrieval/Response.java
Thu Jul 18 14:31:42 CEST 2013
by Michael Peter Christen
added a getSegmentCount method and use it to disable optimize if wanted
current segment count is below optimization level
Changed Files: source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java
Thu Jul 18 13:26:07 CEST 2013
by Michael Peter Christen
- used reger's LukeRequest to generalize the index info in
SolrServerConnector
- used the LukeRequest in SolrServerConnector to replace the index size
method by a getNumDocs request to a LukeRequest result
Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/migration.java
Thu Jul 18 12:29:04 CEST 2013
by Michael Peter Christen
enhancements to crawler
Changed Files: source/net/yacy/crawler/Balancer.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java
Wed Jul 17 19:09:25 CEST 2013
by orbiter
removed double-ocurring value assignments
Changed Files: htroot/api/status_p.java
Mon Jul 15 18:22:35 CEST 2013
by Roland Haeder
Removed trailing spaces + some more final
Changed Files: source/net/yacy/search/index/Fulltext.java
Sun Jul 14 09:01:10 CEST 2013
by Roland Haeder
Added some cora package loggers
Changed Files: defaults/yacy.logging
Sun Jul 14 08:04:10 CEST 2013
by Roland Haeder
Recommended cleanup (please, one day, execute this cleanup)
Changed Files: .settings/org.eclipse.jdt.ui.prefs
Wed Jul 17 16:42:10 CEST 2013
by Felix Ableitner
Fixed (?i) appearing in entries, fixed multiple equal lines in file.
Changed Files: source/net/yacy/repository/Blacklist.java
Wed Jul 17 12:38:58 CEST 2013
by orbiter
logging of deadlocks (if any) during cleanup process
Changed Files: source/net/yacy/search/Switchboard.java
Wed Jul 17 12:38:12 CEST 2013
by orbiter
added cpu load information to status page
Changed Files: htroot/Status.java, htroot/Status_p.inc
Tue Jul 16 17:16:48 CEST 2013
by Michael Peter Christen
do a more frequent optimization, reduces IO after optimization
Changed Files: build.properties, source/net/yacy/search/Switchboard.java
Tue Jul 16 10:49:20 CEST 2013
by orbiter
added load and deadlock detector in Memory util
Changed Files: source/net/yacy/cora/util/Memory.java
Sun Jul 14 18:43:32 CEST 2013
by Lotus
partly revert latest windows changes:
YaCy has to be installed to a directory with write access for the
running user.
DATA folder is now used in the YaCy folder again.
For using another location, the start script has to be heavily modified
for loading proper start parameters after YaCy has been started once.
Changed Files: startYACY.bat, startYACY_debug.bat
Sun Jul 14 10:15:47 CEST 2013
by Michael Peter Christen
grr
Changed Files: source/net/yacy/search/Switchboard.java
Sun Jul 14 10:02:08 CEST 2013
by Michael Peter Christen
automatically running optimize on solr if user/search is idle for some
time
Changed Files: source/net/yacy/search/Switchboard.java
Sat Jul 13 19:09:46 CEST 2013
by orbiter
do a light optimization at the end of a crawl postprocessing
Changed Files: source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java
Sat Jul 13 18:54:09 CEST 2013
by orbiter
replaced type of solr schema object sku of text_en_splitting_tight by
string
Changed Files: defaults/solr/schema.xml, source/net/yacy/search/schema/CollectionSchema.java
Sat Jul 13 15:12:51 CEST 2013
by Roland Haeder
Merge remote-tracking branch 'upstream/master'
Changed Files: htroot/yacysearch.java, source/net/yacy/search/schema/CollectionConfiguration.java
Sat Jul 13 14:49:36 CEST 2013
by orbiter
stealth mode shall only be available for authorized users, because
unauthorized users can otherwise be monitored by authorized users
Changed Files: htroot/yacysearch.java
Fri Jul 12 16:05:48 CEST 2013
by orbiter
activate api actions after postprocessing of crawls
Changed Files: source/net/yacy/search/Switchboard.java
Fri Jul 12 14:29:22 CEST 2013
by orbiter
added a postprocessing field into api/status_p.xml to show if the
postprocessing task is running at that time (status: busy) or not
(status:idle)
Changed Files: htroot/api/status_p.java, htroot/api/status_p.xml, source/net/yacy/search/Switchboard.java
Fri Jul 12 14:16:51 CEST 2013
by orbiter
added list of crawls to status_p.xml
Changed Files: htroot/api/status_p.java, htroot/api/status_p.xml
Thu Jul 11 22:04:01 CEST 2013
by Lotus
added files to uninstall
Changed Files: build.nsi
Thu Jul 11 21:50:06 CEST 2013
by Lotus
Do not allow automatic update in case YaCy is installed to the Program
Files folder on Windows. There are no permissions to write that folder
and update would fail.
Changed Files: htroot/ConfigUpdate_p.html, htroot/ConfigUpdate_p.java
Thu Jul 11 21:48:49 CEST 2013
by Lotus
Revert "Revert "Windows installer: update logo""

This reverts commit c66631d407758b2157001ec5abcf98a2fc36c9ad.
Changed Files: addon/installer/logo.bmp
Thu Jul 11 21:48:13 CEST 2013
by Lotus
Revert "Windows installer: update logo"

This reverts commit 41cc9be62babcea4c4fecb7c0eac8e624f394b70.
Changed Files: addon/installer/logo.bmp
Thu Jul 11 21:46:46 CEST 2013
by Lotus
Windows installer: update logo
Changed Files: addon/installer/logo.bmp
Thu Jul 11 21:46:17 CEST 2013
by Lotus
adapt windows start script parameters to linux start script parameter
Changed Files: startYACY.bat, startYACY_debug.bat
Thu Jul 11 17:04:24 CEST 2013
by orbiter
minimum access time has a tight connection to ClientIdentification,
therefore it is defined there.
Changed Files: htroot/api/latency_p.java, source/net/yacy/cora/protocol/ClientIdentification.java, source/net/yacy/crawler/data/NoticedURL.java
Thu Jul 11 13:10:23 CEST 2013
by Felix Ableitner
Fixed problems with blacklist entry insertion.
Changed Files: htroot/Blacklist_p.java
Thu Jul 11 11:54:11 CEST 2013
by Michael Peter Christen
added Connection:close to all http requests in our http client to
prevent CLOSE_WAIT states (as seen in lsof)
Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java
Wed Jul 10 18:32:12 CEST 2013
by sixcooler
re-insert this file - was deleted by mistake
+ correct an other case-typo
Changed Files: htroot/IndexReIndexMonitor_p.html
Wed Jul 10 15:13:04 CEST 2013
by orbiter
metager whitelist update
Changed Files: defaults/yacy.network.metager.unit
Tue Jul 09 11:42:13 CEST 2013
by orbiter
enhanced the network scanner: find more hosts automatically by removal
of common subdomains before application of protocol-specific prefix
Changed Files: htroot/CrawlStartScanner_p.html, htroot/CrawlStartScanner_p.java
Tue Jul 09 11:01:37 CEST 2013
by orbiter
better handling of time-out at solrj in case that a commit is done in a
fail-over case during add
Changed Files: source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java
Tue Jul 09 10:50:47 CEST 2013
by orbiter
fix for bug #252
the naming of the servlet was wrong, the bug may not be present on
systems where upper/lowercase matching is lazy (windows)
Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/env/templates/submenuIndexControl.template
Mon Jul 08 23:40:31 CEST 2013
by Michael Peter Christen
increase crawler speed by reduction if the busysleep time
Changed Files: defaults/yacy.init
Mon Jul 08 23:32:15 CEST 2013
by Michael Peter Christen
Merge remote-tracking branch 'nutomics/blacklist_structure'
Changed Files: htroot/Blacklist_p.java, source/net/yacy/repository/Blacklist.java
Mon Jul 08 16:29:30 CEST 2013
by Michael Peter Christen
- better concurrency for network scanner
- network scanner can now start from the list of all hosts in the search
index
Changed Files: htroot/CrawlStartScanner_p.html, htroot/CrawlStartScanner_p.java, source/net/yacy/cora/protocol/Scanner.java
Mon Jul 08 16:26:11 CEST 2013
by Michael Peter Christen
fix for citation index generation in case that entry.referrerhash() is
null. This is especially the case if ftp sites are crawled
Changed Files: source/net/yacy/crawler/CrawlStacker.java
Mon Jul 08 16:20:40 CEST 2013
by Michael Peter Christen
accept (but ignore) a '+' sign in front of search words
Changed Files: source/net/yacy/search/query/QueryGoal.java
Mon Jul 08 13:17:09 CEST 2013
by orbiter
enhanced network scanner
- textarea input field can now be used to paste in a large list of hosts
- /31er subnet is possible (only one host)
- auto-detect subdomains for ftp and www subdomains
Changed Files: htroot/CrawlStartScanner_p.html, htroot/CrawlStartScanner_p.java, source/net/yacy/cora/protocol/Scanner.java
Sun Jul 07 19:17:29 CEST 2013
by Lotus
do not cut text on submit buttons
Changed Files: skins/pdblue.css
Thu Jul 04 19:16:13 CEST 2013
by sixcooler
do not use remote proxy if not switched on - regardless of the proto
Changed Files: source/net/yacy/cora/protocol/http/ProxySettings.java
Thu Jul 04 19:13:51 CEST 2013
by sixcooler
Revert "do not use remote proxy if not switched on - regardless of the proto"

This reverts commit 20f452d22809ccb55f177c9b1651f878922e495c.
Changed Files: .gitignore, source/net/yacy/cora/protocol/http/ProxySettings.java
Thu Jul 04 19:12:50 CEST 2013
by sixcooler
do not use remote proxy if not switched on - regardless of the proto
Changed Files: .gitignore, source/net/yacy/cora/protocol/http/ProxySettings.java
Thu Jul 04 19:10:57 CEST 2013
by sixcooler
re-enable saved setting for proxy-crawl-profile
Changed Files: source/net/yacy/crawler/CrawlSwitchboard.java
Thu Jul 04 19:08:53 CEST 2013
by sixcooler
For indexes that are changing rapidly in NRT situations, fcs (stands for
Field Cache per Segment) may be a better choice than the default fc.
(saves memory)
see: http://wiki.apache.org/solr/SimpleFacetParameters#facet.method
Changed Files: source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/search/query/QueryParams.java
Thu Jul 04 18:37:57 CEST 2013
by Felix Ableitner
Changed class structure of Blacklist.
Changed Files: htroot/Blacklist_p.java, source/net/yacy/repository/Blacklist.java
Thu Jul 04 17:12:19 CEST 2013
by Michael Peter Christen
added a patch from Sebastian M.B., submitted by email for coloring of
rss terminal
Changed Files: htroot/rssTerminal.html
Thu Jul 04 16:56:54 CEST 2013
by Michael Peter Christen
Merge commit 'fd90fcc4e08f80acbfd1c9a7ec62ce04cd309594'
Changed Files: htroot/yacysearch.html
Tue Jul 02 20:45:41 CEST 2013
by Felix Ableitner
Fixes #196.
Changed Files: htroot/yacysearch.html
Tue Jul 02 14:29:13 CEST 2013
by Michael Peter Christen
new robots_i attribute fields
Changed Files: defaults/solr.collection.schema, source/net/yacy/search/schema/CollectionConfiguration.java
Mon Jul 01 13:24:52 CEST 2013
by Michael Peter Christen
hack to prevent long waiting times in crawler
Changed Files: source/net/yacy/crawler/Balancer.java
Mon Jul 01 13:10:09 CEST 2013
by Michael Peter Christen
prepartion for greedy crawl profiles and refactoring
Changed Files: htroot/CrawlProfileEditor_p.java, htroot/Crawler_p.java, htroot/IndexCreateQueues_p.java, htroot/WatchWebStructure_p.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/search/Switchboard.java
Mon Jul 01 13:09:41 CEST 2013
by Michael Peter Christen
adding the canonical tag to crawl queues
Changed Files: htroot/solr/select.java, source/net/yacy/document/Document.java, source/net/yacy/search/schema/CollectionConfiguration.java
Sun Jun 30 11:30:39 CEST 2013
by orbiter
updated copyright message; included LGPL for 'cora' and a warranty
warning.
Changed Files: COPYRIGHT
Sun Jun 30 02:11:46 CEST 2013
by Michael Peter Christen
redesign of citation index storage
Changed Files: htroot/HostBrowser.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java
Sat Jun 29 20:28:14 CEST 2013
by orbiter
set crawlingQ to true by default because most webpages are dynamic and
crawlingQ should only be switched off in case of crawler traps
Changed Files: defaults/yacy.init
Sat Jun 29 18:54:30 CEST 2013
by Lotus
windows installer: update to latest jre
Changed Files: build.nsi
Sat Jun 29 00:35:43 CEST 2013
by reger
jpeg parser: extract GPS location from meta data
Changed Files: source/net/yacy/document/parser/images/genericImageParser.java
Fri Jun 28 15:33:19 CEST 2013
by Michael Peter Christen
fixed greedy learning mode - global is not a search attribute in
searchitems
Changed Files: htroot/yacysearchitem.java
Fri Jun 28 15:32:36 CEST 2013
by Michael Peter Christen
removed unused option for queries
Changed Files: htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, htroot/yacysearch.java, source/net/yacy/search/query/QueryParams.java
Fri Jun 28 15:31:36 CEST 2013
by Michael Peter Christen
added configuration option for greedy learning function to ConfigPortal
servlet
Changed Files: htroot/ConfigPortal.html, htroot/ConfigPortal.java
Fri Jun 28 15:19:50 CEST 2013
by sixcooler
possible helpful closing of solr-requests
Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java
Fri Jun 28 14:51:37 CEST 2013
by sixcooler
added optimize-option
Changed Files: htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, source/net/yacy/search/index/Fulltext.java
Fri Jun 28 14:17:14 CEST 2013
by Michael Peter Christen
removed attributes from ConfigPortal.html which are redundant to
ConfigSearchPage_p.html
Changed Files: htroot/ConfigPortal.html, htroot/ConfigPortal.java
Fri Jun 28 12:39:36 CEST 2013
by Michael Peter Christen
fixed false multiple-generation of remote facet search which
caused high cpu usage on remote side.
Changed Files: source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java
Fri Jun 28 12:22:33 CEST 2013
by Michael Peter Christen
- fix to gsa searchresult answer in case that no query part is given
- fix to gsa default number of results (is 'num')
Changed Files: htroot/env/templates/header.template, htroot/gsa/searchresult.java
Wed Jun 26 16:51:36 CEST 2013
by Michael Peter Christen
added url_protocol_s to error documents
Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java
Wed Jun 26 09:27:22 CEST 2013
by Michael Peter Christen
refactoring
Changed Files: htroot/solr/select.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java
Wed Jun 26 09:26:34 CEST 2013
by Michael Peter Christen
added metadata-extractor-2.6.2.jar to eclipse classpath, removed old lib
Changed Files: .classpath, source/net/yacy/document/parser/images/genericImageParser.java
Tue Jun 25 23:24:02 CEST 2013
by reger
bump up lib/metadata-extractor-2.6.2.jar (used for image parser) with needed code adjustments
Changed Files: addon/YaCy.app/Contents/Info.plist, build.xml, lib/metadata-extractor-2.6.2.License, lib/metadata-extractor-2.6.2.jar, source/net/yacy/document/parser/images/genericImageParser.java
Tue Jun 25 16:36:05 CEST 2013
by Michael Peter Christen
added url_file_name_s in qeury for live-search of urls
Changed Files: htroot/HostBrowser.html, htroot/IndexControlURLs_p.html
Sun Jun 23 00:39:15 CEST 2013
by reger
make imageparser fileextension detection case insensitive (extensions are often upper case)
Changed Files: source/net/yacy/document/parser/images/genericImageParser.java
Tue Jun 18 16:51:56 CEST 2013
by Michael Peter Christen
now writing image alt texts and (camelcase-)parsed urls into a text
search field for a better image retrieval
Changed Files: source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionConfiguration.java
Tue Jun 18 16:25:00 CEST 2013
by Michael Peter Christen
added daterange option to gsa api
Changed Files: htroot/gsa/searchresult.java
Tue Jun 18 16:22:06 CEST 2013
by Michael Peter Christen
to avoid confusion, the gsa api is available at /search? and
/searchresult?
Changed Files: source/net/yacy/server/http/HTTPDFileHandler.java
Tue Jun 18 13:28:30 CEST 2013
by orbiter
stub for better image indexing
Changed Files: defaults/solr.collection.schema, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java
Mon Jun 17 13:29:03 CEST 2013
by orbiter
added hint to gsa response writer that the result comes from a yacy peer
Changed Files: source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java
Mon Jun 17 13:28:28 CEST 2013
by orbiter
start of next development cycle with small version number 0.01 (as in
the past)
Changed Files: build.properties