Release 1.68 with a lot of important changes - replaced homebrew httpd with jetty - removed jena - enhanced search quality, remote peer selection, search speed, search completion etc. - removed memory leaks and enhanced running in low-memory and low-CPU environments
Commit | Description |
---|---|
Fri Feb 07 01:20:06 CET 2014 by Michael Peter Christen | removed jena library and all code that depended on jena. When jena was introduced, it was also used for search facets. The generic search facets are now deduced from generic solr fields which makes jena as tool for facet semantics superfluous. Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, defaults/yacy.init, htroot/DictionaryLoader_p.html, htroot/DictionaryLoader_p.java, htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/ViewFile.html, htroot/ViewFile.java, htroot/YMarks.java, htroot/api/yacydoc.html, htroot/api/yacydoc.java, htroot/env/templates/submenuSemantic.template, source/net/yacy/document/LibraryProvider.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/yacy.java |
Fri Jan 24 03:11:25 CET 2014 by reger | prepare for a language navigator works fine to restrict language for local solrSearches. More work needs to be done to make rwi/remote searches respect the modifier.language restriction. Changed Files: htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Wed Jan 22 01:31:22 CET 2014 by sixcooler | bump to httpclient-4.3.2 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/dependencies.txt, lib/httpclient-4.3.2.License, lib/httpclient-4.3.2.jar, lib/httpcore-4.3.1.License, lib/httpcore-4.3.1.jar, lib/httpmime-4.3.2.License, lib/httpmime-4.3.2.jar, nbproject/project.xml |
Tue Jan 21 17:04:22 CET 2014 by sixcooler | configurable max-load values for YaCy-Threads: try lower values on smal systems like a Pi Changed Files: defaults/yacy.init, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, source/net/yacy/kelondro/util/MemoryControl.java, source/net/yacy/kelondro/workflow/AbstractBusyThread.java, source/net/yacy/kelondro/workflow/BusyThread.java, source/net/yacy/kelondro/workflow/InstantBusyThread.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/server/serverSwitch.java |
Mon Jan 20 18:31:46 CET 2014 by Michael Peter Christen | - removed old metadata database and all migration code - refactored all code which uses URIMetadataRow as standard for word hash length and word hash ordering and moved that to the class 'Word', becuase the class URIMetadataRow defined the old metadata data structure and should be superfluous in the future - removed unused methods from URIMetadataRow as preparation for further removal of that class Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexFederated_p.java, htroot/SettingsAck_p.java, htroot/yacy/transferRWI.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/HostQueue.java, source/net/yacy/data/BookmarksDB.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/migration.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SearchEvent.java |
Mon Jan 20 00:58:17 CET 2014 by reger | adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString - since specific heuristic Twitter & Blekko is not longer available or redundant with OpenSearchHeuristic, adjusted ConfigHeuristic to use OpensearchHeuristic settings only. For this the default OSD search target list is made available (copied) by default and the other configs are removed. - the return of QueryGoal.getOriginalQueryString includes the queryModifier, which are held separately in a modifier object, but in most (all) cases just the query term is expected, clarified and renamed it to QueryGoal.getQueryString which returns just the search term (if needed a .getOrigianlQueryString could be implemented in Queryparameters, adding the modifiers) - started to adjust internal html href references from absolute to relative (currently it is mixed). For future development we should prefer relative href targets (less trouble with context aware servlets) Changed Files: defaults/yacy.init, htroot/AccessTracker_p.java, htroot/ConfigHeuristics_p.html, htroot/ConfigHeuristics_p.java, htroot/ConfigNetwork_p.java, htroot/api/timeline.java, htroot/index.html, htroot/index.java, htroot/yacysearch.java, htroot/yacysearchitem.java, htroot/yacysearchtrailer.java, source/net/yacy/http/servlets/GSAsearchServlet.java, source/net/yacy/http/servlets/SolrServlet.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Sun Jan 19 15:21:23 CET 2014 by Michael Peter Christen | do YaCy p2p connections using a timeout-request which covers the http request into a separate thread and ignores the furthure result of a request if that does not answer within the requested time-out. This is a try to solve a problem with the peer-ping, which hangs whenever a peer appears to be dead or blocked. Changed Files: source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/federate/opensearch/SRURSSConnector.java, source/net/yacy/cora/federate/yacy/api/Network.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/data/WorkTables.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/xml/opensearchdescriptionReader.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/search/Switchboard.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/yacy.java |
Fri Jan 17 00:02:23 CET 2014 by reger | implemented DIGEST authentication, which is for remote login more secure as BASIC were pwd is transmitted near clear text (B64enc). This has some implication as RFC 2617 requires and recommends a password hash MD5(user:realm:pwd) for DIGEST. !!! before activating DIGEST you have to reassign all passwords !!! to allow new calculation of the hash - default authentication is still BASIC - configuration at this time only manually in (DATA/settings) or defaults/web.xml (<auth-method> - the realmname is in defaults/yacy.init adminRealm=YaCy-AdminUI - fyi: the realmname is shown on login screen - changing the realm name invalidates all passwords - but for security you are encouraged to do so (as localhostadmin) - implemented to support both, old hashes for BASIC and new hashes for BASIC and DIGEST - to differentiate old / new hash the in Jetty used hash-prefix "MD5:" is used for new pwd-hashes ( "MD5:hash" ) Changed Files: defaults/web.xml, defaults/yacy.init, htroot/ConfigAccounts_p.java, htroot/SettingsAck_p.java, htroot/User.java, source/net/yacy/data/UserDB.java, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyLoginService.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Thu Jan 16 17:27:14 CET 2014 by Michael Peter Christen | remote search peer selection schema change: - all non-dht targets (previously separated into 'robinson' for dht-like queries and 'node' for solr queries) are non 'extra' peers, which are queries using solr - these extra-peers are now selected using a ranking on last-seen, peer-tag-matches, node-peer flags, peer age, and link count. The ranking is done using a weight and a random factor. - the number of extra peers is 50% of the dht peers - the dht peers now exclude too young peers to prevent bad results during strong growth of the network - the number of dht peers (and therefore extra-peers) is reduced when the memory of the peer is low and/or some documents still appear in the indexing-queue. This shall prevent a peer from deadlocks when p2p queries are made in a fast sequence on weak hardware. Changed Files: defaults/yacy.network.freeworld.unit, htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, htroot/yacysearch.java, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/cora/federate/yacy/Distribution.java, source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/Transmission.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SearchEventCache.java |
Fri Jan 10 10:42:47 CET 2014 by reger | making root context (htroot) a WebAppContext - this allows additional features, like servlet configuration via web.xml and many more things. - currently the standard servlets are still configured in the code (so the supplied defaults/web.xml is not realy needed, yet), but could be expanded - lookup for web.xml - 1. in /DATA/SETTINGS then in /defaults Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, defaults/web.xml, lib/jetty-webapp-8.1.14.v20131031.jar, lib/jetty-xml-8.1.14.v20131031.jar, nbproject/project.xml, source/net/yacy/http/Jetty8HttpServerImpl.java |
Wed Jan 08 04:46:21 CET 2014 by reger | exclude terms with " for didYouMean suggestion causes Solr error (and wordindex likely finds suggestion) org.apache.solr.core.SolrCore org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse 'text_t:""d"': Lexical error at line 1, column 12. Encountered: <EOF> after : "" at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:171) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:187) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.query(EmbeddedSolrConnector.java:179) at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector$DocListSearcher.<init>(EmbeddedSolrConnector.java:345) at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getCountByQuery(EmbeddedSolrConnector.java:364) at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getCountByQuery(MirrorSolrConnector.java:326) at net.yacy.cora.federate.solr.connector.ConcurrentUpdateSolrConnector.getCountByQuery(ConcurrentUpdateSolrConnector.java:440) at net.yacy.search.index.Segment.getWordCountGuess(Segment.java:464) at net.yacy.data.DidYouMean.getSuggestions(DidYouMean.java:181) at suggest.respond(suggest.java:73) Changed Files: source/net/yacy/search/index/Segment.java |
Tue Jan 07 21:26:50 CET 2014 by reger | use configured admin-username for api calls - the admin user name can be configured, in apiExec calls the default "admin" username is used. TODO: the bin/apicall.sh script should likely take that into account. Changed Files: htroot/ConfigAppearance_p.java, htroot/ConfigLanguage_p.java, htroot/CrawlStartScanner_p.java, htroot/Table_API_p.java, htroot/sharedBlacklist_p.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/federate/yacy/api/Network.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/utils/loaderThreads.java, source/net/yacy/yacy.java |
Mon Jan 06 08:48:43 CET 2014 by reger | update to Jetty 8.1.14 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/jetty-8.1.14v20131031.License, lib/jetty-client-8.1.14.v20131031.jar, lib/jetty-continuation-8.1.14.v20131031.jar, lib/jetty-http-8.1.14.v20131031.jar, lib/jetty-io-8.1.14.v20131031.jar, lib/jetty-security-8.1.14.v20131031.jar, lib/jetty-server-8.1.14.v20131031.jar, lib/jetty-servlet-8.1.14.v20131031.jar, lib/jetty-servlets-8.1.14.v20131031.jar, lib/jetty-util-8.1.14.v20131031.jar, lib/servlet-api-3.0.jar, nbproject/project.xml |
Sun Jan 05 04:23:44 CET 2014 by Michael Peter Christen | refactoring (usage of constant names for attributes of authentication check) Changed Files: htroot/Collage.java, htroot/ConfigAccounts_p.java, htroot/ConfigBasic.java, htroot/SettingsAck_p.java, htroot/Status.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyLoginService.java, source/net/yacy/migration.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/yacy.java |
Mon Dec 30 20:03:59 CET 2013 by reger | fix "add to blacklist" button style in IndexControlRWIs_p - added default filename filter to select field (as only addition to *.black list is permanent) - modified Blacklist_p header/legend to show all active blacklists (to support understanding that all configured lists are active) - removed obsolete code in Blacklist_p servlet Changed Files: htroot/Blacklist_p.html, htroot/Blacklist_p.java, htroot/IndexControlRWIs_p.html, htroot/IndexControlRWIs_p.java, locales/de.lng, source/net/yacy/peers/Protocol.java, source/net/yacy/search/query/SearchEvent.java |
Sun Dec 29 02:00:44 CET 2013 by reger | remove dependency from old serverCore.java - remaining getPortNr not needed (as current release allows only to set plain integer as port, see ConfigBasic) Changed Files: htroot/Status.java, htroot/opensearchdescription.java, htroot/www/welcome.java, htroot/yacysearch.java, htroot/yacysearch_location.java, source/net/yacy/search/Switchboard.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/serverCore.java, source/net/yacy/yacy.java |
Thu Dec 26 05:07:26 CET 2013 by reger | added Windows Service installer - used Apache procrun as it has a small footprint and comes with a GUI to edit the service settings see http://commons.apache.org/proper/commons-daemon/procrun.html - added the service runner exe file under addon/windowsService as is (without renaming the exe files) - included installYaCyWindowsService.bat and uninstallYaCyWindowsService.bat to main directory - which chooses the native exe according to the processor_architecture Changed Files: addon/windowsService/LICENSE.txt, addon/windowsService/NOTICE.txt, addon/windowsService/RELEASE-NOTES.txt, addon/windowsService/amd64/prunsrv.exe, addon/windowsService/ia64/prunsrv.exe, addon/windowsService/prunmgr.exe, addon/windowsService/prunsrv.exe, installYaCyWindowsService.bat, uninstallYaCyWindowsService.bat |
Thu Dec 26 01:16:53 CET 2013 by reger | upd: to open more external links in new browser-tab Changed Files: htroot/Blacklist_p.html, htroot/Bookmarks.html, htroot/ConfigHeuristics_p.html, htroot/ConfigParser.html, htroot/ConfigUpdate_p.html, htroot/ContentControl_p.html, htroot/CrawlStartExpert_p.html, htroot/DictionaryLoader_p.html, htroot/Help.html, htroot/IndexControlURLs_p.html, htroot/Network.html, htroot/RankingSolr_p.html, htroot/RegexTest.html, htroot/Settings_Http.inc, htroot/Status.html, htroot/Table_API_p.html, htroot/Table_RobotsTxt_p.html, htroot/Triplestore_p.html, htroot/ViewProfile.html, htroot/Vocabulary_p.html, htroot/WatchWebStructure_p.html, htroot/WikiHelp.html, htroot/YMarks.html, htroot/YaCySearchPluginFF.html, htroot/api/yacydoc.html, htroot/api/ymarks/test_import.html, htroot/env/templates/header.template, htroot/env/templates/simpleheader.template, htroot/yacy/ui/index.html, htroot/yacy/ui/yacyui-welcome.html, htroot/yacyinteractive.html, htroot/yacysearch.html, htroot/yacysearch_location.html |
Wed Dec 25 05:20:13 CET 2013 by reger | added SSL/HTTPS connector to support SSL/https connection on port 8443 !!! attention !!! to make sure YaCy can start, https will be disabled if port 8443 is used - added ping test for above to migration - as of now port for https is hardcoded to default 8443 - if not urgend required I'd leave it this way (it's standard) to use different ports for http and https - post https port on ConfigBasic.html (if active) Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/YaCyHttpServer.java, source/net/yacy/migration.java, source/net/yacy/search/Switchboard.java, source/net/yacy/yacy.java |
Mon Dec 23 01:23:40 CET 2013 by Michael Peter Christen | removed unused anomichttpd code after migration to jetty Changed Files: defaults/yacy.init, htroot/AccessTracker_p.java, htroot/Connections_p.java, htroot/yacy/hello.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/YaCyDefaultServlet.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyProxyServlet.java, source/net/yacy/http/YacyDomainHandler.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/serverCore.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/yacy.java |
Thu Dec 19 21:51:05 CET 2013 by Michael Peter Christen | migrated to solr 4.6.0 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, defaults/solr/solrconfig.xml, lib/lucene-analyzers-common-4.6.0.jar, lib/lucene-analyzers-phonetic-4.6.0.jar, lib/lucene-classification-4.6.0.jar, lib/lucene-codecs-4.6.0.jar, lib/lucene-core-4.6.0.jar, lib/lucene-facet-4.6.0.jar, lib/lucene-grouping-4.6.0.jar, lib/lucene-highlighter-4.6.0.jar, lib/lucene-join-4.6.0.jar, lib/lucene-memory-4.6.0.jar, lib/lucene-misc-4.6.0.jar, lib/lucene-queries-4.6.0.jar, lib/lucene-queryparser-4.6.0.jar, lib/lucene-spatial-4.6.0.jar, lib/lucene-suggest-4.6.0.jar, lib/solr-core-4.6.0.jar, lib/solr-solr-4.6.0.License, lib/solr-solrj-4.6.0.License, lib/solr-solrj-4.6.0.jar, source/net/yacy/search/index/Fulltext.java |
Sat Dec 14 20:02:24 CET 2013 by reger | Merge origin/master into jetty Changed Files: build.properties, htroot/SettingsAck_p.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/get_xbel.java, htroot/solr/select.java, htroot/yacy/seedlist.java, nbproject/project.xml, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/data/ymark/YMarkDMOZImporter.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/data/ymark/YMarkUtil.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Fri Dec 13 15:56:29 CET 2013 by Michael Peter Christen | - added a SolrQueryResponse2SolrDocumentList method which is able to work around the unfolding process in Solr's BinaryResponseWriter. This was a huge performance bottleneck in the embedded solr connector and the problem is actually on Solr side, but we have now a workaround. - This made it possible to abstract a high-performance index access method which is implemented as method getDocumentListByParams. That method is also implemented in the SolrServerConnector and provides a very efficient access to a solr index if the index is embedded. - a popular use of the document list retrieval is a result count which can now also make use of the new method, via getDocumentCountByParams. - enhanced the Error cache which now does not store error documents within the ram cache if the document is also written to solr. When documents are retrieved from the cache, they are partly read from the ram cache and if not existent there, from the Solr index. Changed Files: source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Thu Dec 05 22:53:29 CET 2013 by reger | Merge origin/master into jetty Changed Files: addon/YaCy.app/Contents/Info.plist, build.properties, defaults/solr.collection.schema, defaults/solr.webgraph.schema, htroot/ConfigHTCache_p.java, htroot/CrawlStartScanner_p.html, htroot/Crawler_p.java, htroot/IndexControlURLs_p.java, htroot/IndexFederated_p.java, htroot/PerformanceGraph.java, htroot/PerformanceMemory_p.html, htroot/PerformanceMemory_p.java, htroot/Performance_p.html, htroot/api/status_p.java, htroot/api/yacydoc.java, htroot/js/yacyinteractive.js, htroot/yacy/hello.java, htroot/yacysearchitem.java, source/net/yacy/cora/federate/opensearch/OpenSearchConnector.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/graphics/ProfilingGraph.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/snippet/ResultEntry.java, startYACY.bat |
Wed Dec 04 01:54:45 CET 2013 by Michael Peter Christen | enhanced webgraph processing Changed Files: htroot/Crawler_p.java, htroot/IndexControlURLs_p.java, htroot/IndexFederated_p.java, htroot/api/status_p.java, htroot/api/yacydoc.java, source/net/yacy/cora/federate/opensearch/OpenSearchConnector.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/snippet/ResultEntry.java |
Fri Nov 29 22:39:52 CET 2013 by reger | Merge origin/master into jetty Conflicts: source/net/yacy/cora/federate/solr/SolrServlet.java Changed Files: defaults/yacy.init, htroot/HostBrowser.java, htroot/gsa/searchresult.java, htroot/robots.txt, htroot/solr/select.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/document/Condenser.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Sun Nov 24 21:00:39 CET 2013 by reger | Merge origin/master into jetty Changed Files: htroot/Blog.java, htroot/BlogComments.java, htroot/HostBrowser.java, htroot/IndexControlURLs_p.java, htroot/Wiki.java, htroot/portalsearch/yacy-portalsearch.js, htroot/yacy/seedlist.html, htroot/yacy/seedlist.java, htroot/yacy/seedlist.json, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/cora/order/Base64Order.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/snippet/ResultEntry.java |
Sun Nov 17 00:18:25 CET 2013 by reger | Merge origin/master into jetty Changed Files: defaults/solr/solrconfig.xml, defaults/yacy.init, defaults/yacy.logging, htroot/ConfigBasic.java, htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java, htroot/Crawler_p.html, htroot/Crawler_p.java, htroot/User.java, htroot/ViewProfile.java, htroot/api/status_p.java, htroot/api/status_p.xml, htroot/api/termlist_p.java, htroot/api/webstructure.java, htroot/gsa/searchresult.java, htroot/index.java, htroot/js/Crawler.js, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/language/synonyms/SynonymLibrary.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/storage/HandleSet.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/document/Document.java, source/net/yacy/document/language/LanguageStatistics.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/serverObjects.java, source/net/yacy/utils/gzip.java, source/net/yacy/visualization/RasterPlotter.java |
Wed Nov 13 13:38:01 CET 2013 by Michael Peter Christen | added option to Config_Network_p.html to enable remote search while DHT-Receive is switched off. Changed Files: defaults/yacy.init, htroot/ConfigBasic.java, htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java, htroot/index.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/server/serverObjects.java |
Wed Nov 13 06:18:48 CET 2013 by Michael Peter Christen | set more logger to 'final static' Changed Files: htroot/api/termlist_p.java, source/net/yacy/cora/language/synonyms/SynonymLibrary.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/document/language/LanguageStatistics.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/utils/gzip.java |
Thu Nov 07 21:30:17 CET 2013 by reger | merge rc1/master Changed Files: defaults/solr/solrconfig.xml, defaults/yacy.init, htroot/ConfigHTCache_p.html, htroot/ConfigHTCache_p.java, htroot/ContentAnalysis_p.java, htroot/HostBrowser.java, htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/RankingSolr_p.java, htroot/yacysearch.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/data/BookmarksDB.java, source/net/yacy/document/parser/html/CharacterCoding.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/http/CrashProtectionHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/SSIHandler.java, source/net/yacy/http/TemplateHandler.java, source/net/yacy/http/YaCyHttpServer.java, source/net/yacy/peers/Transmission.java, source/net/yacy/search/ResourceObserver.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Thu Nov 07 10:01:44 CET 2013 by Michael Peter Christen | found and fixed a huge memory leak in solr caching (inside Solr). The not-flushed Solr cache is now handled in this way: - it is smaller by default - an Solr-internal process is started to flush the cache periodically (this does NOT clean the cache, just removes old objects) - a Solr-external process (the standard YaCy cleanup-process) now has direct access to the solr internal cache and flushes them completely. The time frame for such a flush is defined by the cleanup-process frequency, by default 10 minutes. Changed Files: defaults/solr/solrconfig.xml, htroot/ContentAnalysis_p.java, htroot/RankingSolr_p.java, htroot/yacysearch.java, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/search/ResourceObserver.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java |
Sat Nov 02 00:48:42 CET 2013 by reger | Merge origin/master into jetty Changed Files: build.properties, htroot/HostBrowser.java, libbuild/GitRevTask/GitRevTask.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Fri Nov 01 17:24:36 CET 2013 by Michael Peter Christen | Speed enhancements and less CPU usage during Solr searches when using the embedded Solr (the default). This was obtained by cirumventing solrj search encapsulation and the implementation of direct index access methods to Solr. The effect will not only be seen during search, but this has also a strong effect on suggestions (much more) and less CPU power usage during index distribution (which needs many search requests) Changed Files: source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Sun Oct 27 15:09:24 CET 2013 by reger | Merge origin/master into jetty Changed Files: build.properties, defaults/yacy.init, defaults/yacy.logging, htroot/CacheResource_p.java, htroot/ConfigProperties_p.html, htroot/ConfigSearchPage_p.html, htroot/ConfigUpdate_p.html, htroot/CrawlCheck_p.java, htroot/CrawlResults.java, htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java, htroot/Crawler_p.java, htroot/IndexControlURLs_p.java, htroot/Table_API_p.html, htroot/Table_API_p.java, htroot/ViewFile.java, htroot/ViewLog_p.java, htroot/compare_yacy.html, htroot/compare_yacy.java, htroot/env/base.css, htroot/env/grafics/rss.png, htroot/opensearchdescription.java, htroot/opensearchdescription.xml, htroot/yacy/ui/js/jquery.tagcloud.js, htroot/yacysearch.html, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, source/net/yacy/cora/document/analysis/Classification.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/order/Digest.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkMetadata.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/SentenceReader.java, source/net/yacy/document/WordTokenizer.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/logging/GuiHandler.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/server/http/HTTPDFileHandler.java |
Fri Oct 25 15:05:30 CEST 2013 by Michael Peter Christen | hacks to prevent storage of data longer than necessary during search and some speed enhancements. This should reduce the memory usage during heavy-load search a bit. Changed Files: defaults/yacy.init, htroot/ViewFile.java, source/net/yacy/cora/order/Digest.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/SentenceReader.java, source/net/yacy/document/WordTokenizer.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/snippet/TextSnippet.java |
Wed Oct 23 00:16:54 CEST 2013 by Michael Peter Christen | - fixed a problem which ocurred when a document was not recognized with the right content domain (i.e. identifying that it is an image, text etc.) because it used the file extension and not an existing mime type assignment. - fixed the new setting that images shall be loaded for a better image search. - both fixes together makes it now possible to crawl commons.wikimedia.org which makes use of 'funny' document names (i.e. ending with .jpg while the document is html) Changed Files: defaults/yacy.init, htroot/CrawlStartExpert_p.java, source/net/yacy/cora/document/analysis/Classification.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/data/ymark/YMarkMetadata.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/SearchEvent.java |
Wed Oct 16 03:02:21 CEST 2013 by reger | merge rc1/master Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, htroot/Blacklist_p.java, htroot/BlogComments.java, htroot/Connections_p.java, htroot/Status.java, htroot/User.java, htroot/Wiki.java, htroot/sharedBlacklist_p.java, lib/dependencies.txt, lib/httpclient-4.3.1.License, lib/httpclient-4.3.1.jar, lib/httpmime-4.3.1.License, lib/httpmime-4.3.1.jar, nbproject/project.xml, source/net/yacy/server/http/HTTPDProxyHandler.java |
Fri Oct 11 01:50:25 CEST 2013 by reger | merge rc1/master Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, addon/synonyms/LICENSE.txt, addon/synonyms/openthesaurus_de_yacy, build.xml, defaults/solr.collection.schema, defaults/solr/solrconfig.xml, defaults/yacy.init, htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java, htroot/IndexControlRWIs_p.java, htroot/RankingSolr_p.html, htroot/RankingSolr_p.java, htroot/Table_API_p.html, htroot/Table_API_p.java, htroot/env/grafics/doc.gif, htroot/env/grafics/right.gif, htroot/yacy/search.java, htroot/yacysearch.java, lib/lucene-analyzers-common-4.5.0.jar, lib/lucene-analyzers-phonetic-4.5.0.jar, lib/lucene-classification-4.5.0.jar, lib/lucene-codecs-4.5.0.jar, lib/lucene-core-4.5.0.jar, lib/lucene-facet-4.5.0.jar, lib/lucene-grouping-4.5.0.jar, lib/lucene-highlighter-4.5.0.jar, lib/lucene-join-4.5.0.jar, lib/lucene-memory-4.5.0.jar, lib/lucene-misc-4.5.0.jar, lib/lucene-queries-4.5.0.jar, lib/lucene-queryparser-4.5.0.jar, lib/lucene-spatial-4.5.0.jar, lib/lucene-suggest-4.5.0.jar, lib/solr-core-4.5.0.jar, lib/solr-solrj-4.5.0.License, lib/solr-solrj-4.5.0.jar, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/data/DidYouMean.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/server/http/HTTPDProxyHandler.java |
Wed Oct 09 20:49:14 CEST 2013 by Jens Bertram | Merge upstream/master into crawlexpert-post Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, addon/synonyms/LICENSE.txt, addon/synonyms/openthesaurus_de_yacy, build.xml, defaults/solr.collection.schema, defaults/solr/solrconfig.xml, htroot/IndexControlRWIs_p.java, htroot/RankingSolr_p.html, htroot/RankingSolr_p.java, htroot/yacy/search.java, htroot/yacysearch.java, lib/lucene-analyzers-common-4.5.0.jar, lib/lucene-analyzers-phonetic-4.5.0.jar, lib/lucene-classification-4.5.0.jar, lib/lucene-codecs-4.5.0.jar, lib/lucene-core-4.5.0.jar, lib/lucene-facet-4.5.0.jar, lib/lucene-grouping-4.5.0.jar, lib/lucene-highlighter-4.5.0.jar, lib/lucene-join-4.5.0.jar, lib/lucene-memory-4.5.0.jar, lib/lucene-misc-4.5.0.jar, lib/lucene-queries-4.5.0.jar, lib/lucene-queryparser-4.5.0.jar, lib/lucene-spatial-4.5.0.jar, lib/lucene-suggest-4.5.0.jar, lib/solr-core-4.5.0.jar, lib/solr-solrj-4.5.0.License, lib/solr-solrj-4.5.0.jar, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Wed Oct 09 14:39:20 CEST 2013 by Jens Bertram | Merge master into crawlexpert-post Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, addon/synonyms/LICENSE.txt, addon/synonyms/openthesaurus_de_yacy, build.xml, defaults/solr.collection.schema, defaults/solr/solrconfig.xml, htroot/IndexControlRWIs_p.java, htroot/RankingSolr_p.html, htroot/RankingSolr_p.java, htroot/yacy/search.java, htroot/yacysearch.java, lib/lucene-analyzers-common-4.5.0.jar, lib/lucene-analyzers-phonetic-4.5.0.jar, lib/lucene-classification-4.5.0.jar, lib/lucene-codecs-4.5.0.jar, lib/lucene-core-4.5.0.jar, lib/lucene-facet-4.5.0.jar, lib/lucene-grouping-4.5.0.jar, lib/lucene-highlighter-4.5.0.jar, lib/lucene-join-4.5.0.jar, lib/lucene-memory-4.5.0.jar, lib/lucene-misc-4.5.0.jar, lib/lucene-queries-4.5.0.jar, lib/lucene-queryparser-4.5.0.jar, lib/lucene-spatial-4.5.0.jar, lib/lucene-suggest-4.5.0.jar, lib/solr-core-4.5.0.jar, lib/solr-solrj-4.5.0.License, lib/solr-solrj-4.5.0.jar, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Wed Oct 09 00:40:48 CEST 2013 by reger | downgrade to Jetty 8 to assure support of JRE 1.6 - introduce a YaCyHttp interface to modulize/separate http server - adjust the Jetty version specific implementation part (in package net.yacy.http) - putting the version specific code in classes starting with Jetty8xxxx - moved existing Jetty9xxx implementation into a test class (to keep the code) - adjust build to the changed jars - make use of the introduced YaCyHttpServer interface in related htroot servlets - adjust other test cases/classes Changed Files: build.xml, htroot/ConfigBasic.java, htroot/PerformanceQueues_p.java, htroot/SettingsAck_p.java, htroot/Status.java, lib/jetty-continuation-8.1.13.v20130916.jar, lib/jetty-http-8.1.13.v20130916.jar, lib/jetty-io-8.1.13.v20130916.jar, lib/jetty-security-8.1.13.v20130916.jar, lib/jetty-server-8.1.13.v20130916.jar, lib/jetty-servlet-8.1.13.v20130916.jar, lib/jetty-servlets-8.1.13.v20130916.jar, lib/jetty-util-8.1.13.v20130916.jar, libt/jetty-continuation-9.0.5.v20130815.jar, libt/jetty-http-9.0.5.v20130815.jar, libt/jetty-io-9.0.5.v20130815.jar, libt/jetty-security-9.0.5.v20130815.jar, libt/jetty-server-9.0.5.v20130815.jar, libt/jetty-servlet-9.0.5.v20130815.jar, libt/jetty-servlets-9.0.5.v20130815.jar, libt/jetty-util-9.0.5.v20130815.jar, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/http/YaCyDefaultServlet.java, source/net/yacy/http/YaCyHttpServer.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/yacy.java, test/de/anomic/document/ParserTest.java, test/de/anomic/yacy/yacyURLTest.java, test/net/yacy/cora/document/MultiProtocolURITest.java, test/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnectorTest.java, test/net/yacy/http/Jetty9HttpServerImpl.java, test/net/yacy/http/Jetty9YaCyDefaultServlet.java, test/net/yacy/http/Jetty9YaCySecurityHandler.java, test/net/yacy/http/Jetty9_Implementation_Test.java, test/net/yacy/http/YacyMain.java |
Mon Oct 07 17:09:40 CEST 2013 by Michael Peter Christen | migration to Solr 4.5.0 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, defaults/solr/solrconfig.xml, lib/lucene-analyzers-common-4.5.0.jar, lib/lucene-analyzers-phonetic-4.5.0.jar, lib/lucene-classification-4.5.0.jar, lib/lucene-codecs-4.5.0.jar, lib/lucene-core-4.5.0.jar, lib/lucene-facet-4.5.0.jar, lib/lucene-grouping-4.5.0.jar, lib/lucene-highlighter-4.5.0.jar, lib/lucene-join-4.5.0.jar, lib/lucene-memory-4.5.0.jar, lib/lucene-misc-4.5.0.jar, lib/lucene-queries-4.5.0.jar, lib/lucene-queryparser-4.5.0.jar, lib/lucene-spatial-4.5.0.jar, lib/lucene-suggest-4.5.0.jar, lib/solr-core-4.5.0.jar, lib/solr-solrj-4.5.0.License, lib/solr-solrj-4.5.0.jar, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/schema/CollectionSchema.java |
Mon Sep 30 03:46:39 CEST 2013 by reger | merge with rc1/master Changed Files: addon/YaCy.app/Contents/Info.plist, defaults/solr.collection.schema, defaults/solr.webgraph.schema, defaults/yacy.init, htroot/CrawlProfileEditor_p.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/IndexControlRWIs_p.java, htroot/IndexCreateQueues_p.java, htroot/yacy/search.java, htroot/yacysearch.java, htroot/yacysearch_location.java, htroot/yacysearchtrailer.java, lib/commons-logging-1.1.3.License, lib/commons-logging-1.1.3.jar, lib/dependencies.txt, lib/httpclient-4.3.License, lib/httpclient-4.3.jar, lib/httpcore-4.3.License, lib/httpcore-4.3.jar, lib/httpmime-4.3.License, lib/httpmime-4.3.jar, nbproject/project.xml, source/net/yacy/cora/document/encoding/UTF8.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/cora/federate/solr/Ranking.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/language/synonyms/AutotaggingLibrary.java, source/net/yacy/cora/protocol/ByteArrayBody.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/ProxySettings.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/HostQueue.java, source/net/yacy/crawler/HostQueues.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/peers/Protocol.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java, source/net/yacy/server/http/TemplateEngine.java, source/net/yacy/server/serverSwitch.java |
Sat Sep 28 15:46:59 CEST 2013 by orbiter | Merge remote-tracking branch 'origin/master' Conflicts: .classpath Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, defaults/solr.collection.schema, defaults/solr.webgraph.schema, htroot/CrawlProfileEditor_p.java, htroot/Crawler_p.java, htroot/IndexCreateQueues_p.java, lib/commons-logging-1.1.3.License, lib/commons-logging-1.1.3.jar, lib/dependencies.txt, lib/httpclient-4.3.License, lib/httpclient-4.3.jar, lib/httpcore-4.3.License, lib/httpcore-4.3.jar, lib/httpmime-4.3.License, lib/httpmime-4.3.jar, nbproject/project.xml, source/net/yacy/cora/document/encoding/UTF8.java, source/net/yacy/cora/federate/solr/Ranking.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/ByteArrayBody.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/ProxySettings.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java, source/net/yacy/server/serverSwitch.java |
Wed Sep 25 18:27:54 CEST 2013 by Michael Peter Christen | self-healing of mistakenly deactivated crawl profiles. This fixes a bug which can happen in rare cases when a crawl start and a cleanup process happen at the same time. Changed Files: source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java |
Wed Sep 25 14:48:37 CEST 2013 by sixcooler | bump to httpClient-4.3 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/commons-logging-1.1.3.License, lib/commons-logging-1.1.3.jar, lib/dependencies.txt, lib/httpclient-4.3.License, lib/httpclient-4.3.jar, lib/httpcore-4.3.License, lib/httpcore-4.3.jar, lib/httpmime-4.3.License, lib/httpmime-4.3.jar, nbproject/project.xml, source/net/yacy/cora/document/encoding/UTF8.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/ByteArrayBody.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/ProxySettings.java |
Wed Sep 25 14:38:24 CEST 2013 by Michael Peter Christen | added the new field harvestkey_s to the collection index and the webgraph index which is temporary filled with the crawl profile key. This is used to select a set of documents for post-processing as soon as a crawl is finished. Now the postprocessing for a specific crawl is started when that specific crawl is finished and not at the end of all post-processing steps. Changed Files: defaults/solr.collection.schema, defaults/solr.webgraph.schema, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java |
Sun Sep 22 02:21:24 CEST 2013 by reger | merge rc1 master Changed Files: htroot/Crawler_p.java, htroot/HostBrowser.java, htroot/IndexCreateParserErrors_p.html, htroot/IndexCreateParserErrors_p.java, htroot/QuickCrawlLink_p.java, htroot/yacy/crawlReceipt.java, htroot/yacy/search.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/net/yacy/cora/federate/solr/FailCategory.java, source/net/yacy/cora/language/synonyms/SynonymLibrary.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/yacy.java |
Sun Sep 22 02:09:02 CEST 2013 by reger | Merge remote-tracking branch 'origin/master' into jetty Changed Files: .gitignore, defaults/solr.webgraph.schema, htroot/BlacklistTest_p.java, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/CacheResource_p.java, htroot/Collage.java, htroot/ConfigAppearance_p.java, htroot/ConfigHeuristics_p.java, htroot/ConfigLanguage_p.java, htroot/ConfigNetwork_p.java, htroot/ConfigPortal.java, htroot/ConfigUpdate_p.java, htroot/CrawlCheck_p.java, htroot/CrawlResults.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/HostBrowser.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexCreateLoaderQueue_p.java, htroot/IndexCreateParserErrors_p.java, htroot/IndexCreateQueues_p.java, htroot/IndexDeletion_p.java, htroot/IndexFederated_p.java, htroot/IndexImportOAIPMH_p.java, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Messages_p.java, htroot/QuickCrawlLink_p.java, htroot/ServerScannerList.java, htroot/SettingsAck_p.java, htroot/Supporter.java, htroot/Surftips.java, htroot/Table_API_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.java, htroot/Triple_p.java, htroot/ViewFile.java, htroot/ViewImage.java, htroot/Vocabulary_p.java, htroot/WatchWebStructure_p.java, htroot/WebStructurePicture_p.java, htroot/Wiki.java, htroot/YMarks.java, htroot/api/bookmarks/posts/delete_p.java, htroot/api/citation.java, htroot/api/feed.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/table_p.java, htroot/api/termlist_p.java, htroot/api/webstructure.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/get_xbel.java, htroot/api/ymarks/get_ymark.java, htroot/api/ymarks/import_ymark.java, htroot/cytag.java, htroot/gsa/searchresult.java, htroot/interaction/GetRDF.java, htroot/mediawiki_p.java, htroot/rct_p.java, htroot/sharedBlacklist_p.java, htroot/solr/select.java, htroot/yacy/crawlReceipt.java, htroot/yacy/idx.java, htroot/yacy/message.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacysearch.java, htroot/yacysearch_location.java, htroot/yacysearchitem.java, htroot/yacysearchtrailer.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/document/analysis/Classification.java, source/net/yacy/cora/document/encoding/ASCII.java, source/net/yacy/cora/document/encoding/UTF8.java, source/net/yacy/cora/document/feed/Channel.java, source/net/yacy/cora/document/feed/Channels.java, source/net/yacy/cora/document/feed/Hit.java, source/net/yacy/cora/document/feed/RSSFeed.java, source/net/yacy/cora/document/feed/RSSMessage.java, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/cora/document/id/AnchorURL.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/document/id/Punycode.java, source/net/yacy/cora/federate/opensearch/SRURSSConnector.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ShardSelection.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/cora/federate/yacy/Distribution.java, source/net/yacy/cora/federate/yacy/Peer.java, source/net/yacy/cora/federate/yacy/Peers.java, source/net/yacy/cora/geo/GeoLocation.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/Literal.java, source/net/yacy/cora/lod/Node.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/YaCyMetadata.java, source/net/yacy/cora/order/Base64Order.java, source/net/yacy/cora/order/Digest.java, source/net/yacy/cora/order/StringOrder.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/LinkExtractor.java, source/net/yacy/cora/storage/AbstractMapStore.java, source/net/yacy/cora/storage/KeyList.java, source/net/yacy/cora/util/ByteArray.java, source/net/yacy/cora/util/ByteBuffer.java, source/net/yacy/cora/util/JSONArray.java, source/net/yacy/cora/util/JSONException.java, source/net/yacy/cora/util/JSONObject.java, source/net/yacy/cora/util/JSONTokener.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlQueue.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/crawler/data/ResultImages.java, source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/crawler/retrieval/Request.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/crawler/retrieval/SitemapImporter.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/crawler/robots/RobotsTxtEntry.java, source/net/yacy/crawler/robots/RobotsTxtParser.java, source/net/yacy/data/BlogBoard.java, source/net/yacy/data/BlogBoardComments.java, source/net/yacy/data/BookmarkDate.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/data/BookmarksDB.java, source/net/yacy/data/MessageBoard.java, source/net/yacy/data/URLLicense.java, source/net/yacy/data/UserDB.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/wiki/WikiBoard.java, source/net/yacy/data/ymark/TablesRowComparator.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/data/ymark/YMarkDate.java, source/net/yacy/data/ymark/YMarkEntry.java, source/net/yacy/data/ymark/YMarkMetadata.java, source/net/yacy/data/ymark/YMarkRDF.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/data/ymark/YMarkUtil.java, source/net/yacy/dbtest.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/Parser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/bzipParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/gzipParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ContentTransformer.java, source/net/yacy/document/parser/html/EmbedEntry.java, source/net/yacy/document/parser/html/ImageEntry.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BEncodedHeapBag.java, source/net/yacy/kelondro/blob/BEncodedHeapShard.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapColumnIndex.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/blob/TablesColumnIndex.java, source/net/yacy/kelondro/data/citation/CitationReference.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/navigation/NavigationReferenceRow.java, source/net/yacy/kelondro/data/navigation/NavigationReferenceVars.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/CachedRecords.java, source/net/yacy/kelondro/io/CharBuffer.java, source/net/yacy/kelondro/io/Records.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SQLTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/BDecoder.java, source/net/yacy/kelondro/util/BEncoder.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/Accessible.java, source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/EventChannel.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/Transmission.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/peers/operation/yacyUpdateLocation.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/server/http/AugmentedHtmlStream.java, source/net/yacy/server/http/ChunkedOutputStream.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/http/ServerSideIncludes.java, source/net/yacy/server/http/TemplateEngine.java, source/net/yacy/server/serverCore.java, source/net/yacy/server/serverObjects.java, source/net/yacy/utils/bitfield.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/utils/gzip.java, source/net/yacy/utils/loaderThreads.java |
Tue Sep 17 15:27:02 CEST 2013 by Michael Peter Christen | - removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory Changed Files: htroot/Crawler_p.java, htroot/HostBrowser.java, htroot/IndexCreateParserErrors_p.html, htroot/IndexCreateParserErrors_p.java, htroot/QuickCrawlLink_p.java, htroot/yacy/crawlReceipt.java, htroot/yacy/search.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/net/yacy/cora/federate/solr/FailCategory.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/snippet/MediaSnippet.java |
Sun Sep 15 23:27:04 CEST 2013 by Michael Peter Christen | - replaced the properties object in AnchorURL with distinct variables for anchor attributes. - this caused that large portions of the parser code had to be adopted as well - added a counter target_order_i for anchor links in webgraph computation Changed Files: defaults/solr.webgraph.schema, htroot/ViewFile.java, source/net/yacy/cora/document/id/AnchorURL.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/ResultImages.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/Parser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/bzipParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/gzipParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ImageEntry.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java, source/net/yacy/search/snippet/MediaSnippet.java |
Sun Sep 15 00:30:23 CEST 2013 by Michael Peter Christen | - the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring Changed Files: .gitignore, htroot/BlacklistTest_p.java, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/CacheResource_p.java, htroot/Collage.java, htroot/ConfigAppearance_p.java, htroot/ConfigHeuristics_p.java, htroot/ConfigLanguage_p.java, htroot/ConfigNetwork_p.java, htroot/ConfigPortal.java, htroot/ConfigUpdate_p.java, htroot/CrawlCheck_p.java, htroot/CrawlResults.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/HostBrowser.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexCreateLoaderQueue_p.java, htroot/IndexCreateParserErrors_p.java, htroot/IndexCreateQueues_p.java, htroot/IndexDeletion_p.java, htroot/IndexFederated_p.java, htroot/IndexImportOAIPMH_p.java, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Messages_p.java, htroot/QuickCrawlLink_p.java, htroot/ServerScannerList.java, htroot/SettingsAck_p.java, htroot/Supporter.java, htroot/Surftips.java, htroot/Table_API_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.java, htroot/Triple_p.java, htroot/ViewFile.java, htroot/ViewImage.java, htroot/Vocabulary_p.java, htroot/WatchWebStructure_p.java, htroot/WebStructurePicture_p.java, htroot/Wiki.java, htroot/YMarks.java, htroot/api/bookmarks/posts/delete_p.java, htroot/api/citation.java, htroot/api/feed.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/table_p.java, htroot/api/termlist_p.java, htroot/api/webstructure.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/get_xbel.java, htroot/api/ymarks/get_ymark.java, htroot/api/ymarks/import_ymark.java, htroot/cytag.java, htroot/gsa/searchresult.java, htroot/interaction/GetRDF.java, htroot/mediawiki_p.java, htroot/rct_p.java, htroot/sharedBlacklist_p.java, htroot/solr/select.java, htroot/yacy/crawlReceipt.java, htroot/yacy/idx.java, htroot/yacy/message.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacysearch.java, htroot/yacysearch_location.java, htroot/yacysearchitem.java, htroot/yacysearchtrailer.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/document/analysis/Classification.java, source/net/yacy/cora/document/encoding/ASCII.java, source/net/yacy/cora/document/encoding/UTF8.java, source/net/yacy/cora/document/feed/Channel.java, source/net/yacy/cora/document/feed/Channels.java, source/net/yacy/cora/document/feed/Hit.java, source/net/yacy/cora/document/feed/RSSFeed.java, source/net/yacy/cora/document/feed/RSSMessage.java, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/cora/document/id/AnchorURL.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/document/id/Punycode.java, source/net/yacy/cora/federate/opensearch/SRURSSConnector.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ShardSelection.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/cora/federate/yacy/Distribution.java, source/net/yacy/cora/federate/yacy/Peer.java, source/net/yacy/cora/federate/yacy/Peers.java, source/net/yacy/cora/geo/GeoLocation.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/Literal.java, source/net/yacy/cora/lod/Node.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/YaCyMetadata.java, source/net/yacy/cora/order/Base64Order.java, source/net/yacy/cora/order/Digest.java, source/net/yacy/cora/order/StringOrder.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/LinkExtractor.java, source/net/yacy/cora/storage/AbstractMapStore.java, source/net/yacy/cora/storage/KeyList.java, source/net/yacy/cora/util/ByteArray.java, source/net/yacy/cora/util/ByteBuffer.java, source/net/yacy/cora/util/JSONArray.java, source/net/yacy/cora/util/JSONException.java, source/net/yacy/cora/util/JSONObject.java, source/net/yacy/cora/util/JSONTokener.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlQueue.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/crawler/data/ResultImages.java, source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/crawler/retrieval/Request.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/crawler/retrieval/SitemapImporter.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/crawler/robots/RobotsTxtEntry.java, source/net/yacy/crawler/robots/RobotsTxtParser.java, source/net/yacy/data/BlogBoard.java, source/net/yacy/data/BlogBoardComments.java, source/net/yacy/data/BookmarkDate.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/data/BookmarksDB.java, source/net/yacy/data/MessageBoard.java, source/net/yacy/data/URLLicense.java, source/net/yacy/data/UserDB.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/wiki/WikiBoard.java, source/net/yacy/data/ymark/TablesRowComparator.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/data/ymark/YMarkDate.java, source/net/yacy/data/ymark/YMarkEntry.java, source/net/yacy/data/ymark/YMarkMetadata.java, source/net/yacy/data/ymark/YMarkRDF.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/data/ymark/YMarkUtil.java, source/net/yacy/dbtest.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/Parser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/bzipParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/gzipParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ContentTransformer.java, source/net/yacy/document/parser/html/EmbedEntry.java, source/net/yacy/document/parser/html/ImageEntry.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BEncodedHeapBag.java, source/net/yacy/kelondro/blob/BEncodedHeapShard.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapColumnIndex.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/blob/TablesColumnIndex.java, source/net/yacy/kelondro/data/citation/CitationReference.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/navigation/NavigationReferenceRow.java, source/net/yacy/kelondro/data/navigation/NavigationReferenceVars.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/CachedRecords.java, source/net/yacy/kelondro/io/CharBuffer.java, source/net/yacy/kelondro/io/Records.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SQLTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/BDecoder.java, source/net/yacy/kelondro/util/BEncoder.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/Accessible.java, source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/EventChannel.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/Transmission.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/peers/operation/yacyUpdateLocation.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/server/http/AugmentedHtmlStream.java, source/net/yacy/server/http/ChunkedOutputStream.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/http/ServerSideIncludes.java, source/net/yacy/server/http/TemplateEngine.java, source/net/yacy/server/serverCore.java, source/net/yacy/server/serverObjects.java, source/net/yacy/utils/bitfield.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/utils/gzip.java, source/net/yacy/utils/loaderThreads.java |
Sat Sep 14 20:49:05 CEST 2013 by reger | update to Jetty 9 jars - include javax.servlet 3.0 Changed Files: build.xml, lib/javax.servlet-3.0.0.v201112011016.jar, lib/jetty-continuation-9.0.5.v20130815.jar, lib/jetty-http-9.0.5.v20130815.jar, lib/jetty-io-9.0.5.v20130815.jar, lib/jetty-security-9.0.5.v20130815.jar, lib/jetty-server-9.0.5.v20130815.jar, lib/jetty-servlet-9.0.5.v20130815.jar, lib/jetty-servlets-9.0.5.v20130815.jar, lib/jetty-util-9.0.5.v20130815.jar, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/http/CrashProtectionHandler.java, source/net/yacy/http/HttpServer.java, source/net/yacy/http/ProxyCacheHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/TemplateHandler.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyLoginService.java, source/net/yacy/http/YaCySecurityHandler.java, source/net/yacy/yacy.java, test/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnectorTest.java |
Tue Sep 10 20:42:38 CEST 2013 by reger | Merge remote-tracking branch 'origin/master' into jetty Changed Files: htroot/gsa/searchresult.java, source/net/yacy/document/Document.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/visualization/AnimationPlotter.java, source/net/yacy/visualization/HexGridPlotter.java, source/net/yacy/visualization/RasterPlotter.java, test/de/anomic/document/ParserTest.java |
Tue Sep 10 10:31:57 CEST 2013 by Michael Peter Christen | added parsing of 'date', 'dc:date', 'dc.date' and 'last-modified' in html meta fields to get a correct (or: better) date timestamp. The http:last-modified mostly does not work because it is set to the current date from most CMS. Changed Files: htroot/gsa/searchresult.java, source/net/yacy/document/Document.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Mon Sep 09 02:36:06 CEST 2013 by reger | merged current rc1/master into jetty branch to allow further development with latest version ServerSideIncludes and servlet return values need further work (for working jetty integration) - TODO: added nasty quickfix to allow SSI - needs further work - TODO: YaCy servlet return values/parameters are not handled Changed Files: .classpath, .gitignore, .settings/org.eclipse.jdt.core.prefs, .settings/org.eclipse.jdt.ui.prefs, COPYRIGHT, addon/YaCy.app/Contents/Info.plist, addon/YaCy.app/Contents/MacOS/JavaApplicationStub, addon/YaCy.app/Contents/Resources/YaCy_2013_Icon.icns, addon/YaCy.ico, addon/YaCy_TrayIcon.png, addon/installer/logo.bmp, addon/testkeys, addon/yacy-svn-4.spec, addon/yacyInit.m4, assembly.xml, bin/addrss.sh, bin/apicall.sh, bin/clearall.sh, bin/clearapi.sh, bin/clearcache.sh, bin/clearindex.sh, bin/indexdump.sh, bin/indexrestore.sh, bin/search.sh, bin/search1.sh, bin/searchall.sh, bin/searchall1.sh, bin/searchtest.sh, bin/up.sh, build.nsi, build.properties, build.xml, debian/changelog, debian/control, debian/postinst, defaults/freeworldKeystore, defaults/heuristicopensearch.conf, defaults/httpd.mime, defaults/solr.collection.schema, defaults/solr.webgraph.schema, defaults/solr/schema.xml, defaults/solr/solr.xml, defaults/solr/solrconfig.xml, defaults/solr/solrcore.properties, defaults/solr/solrcore.x86.properties, defaults/solr/xslt/example.xsl, defaults/solr/xslt/json.xsl, defaults/yacy.init, defaults/yacy.logging, defaults/yacy.network.allip.unit, defaults/yacy.network.freeworld.unit, defaults/yacy.network.intranet.unit, defaults/yacy.network.metager.unit, defaults/yacy.network.webportal.unit, htroot/AccessGrid_p.java, htroot/AccessPicture_p.java, htroot/AccessTracker_p.html, htroot/AccessTracker_p.java, htroot/AugmentedBrowsingFilters_p.java, htroot/AugmentedBrowsing_p.java, htroot/AugmentedParsing_p.java, htroot/Banner.java, htroot/BlacklistCleaner_p.java, htroot/BlacklistImpExp_p.html, htroot/BlacklistImpExp_p.java, htroot/BlacklistTest_p.html, htroot/BlacklistTest_p.java, htroot/Blacklist_p.html, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/CacheResource_p.java, htroot/Collage.java, htroot/ConfigAccounts_p.html, htroot/ConfigAccounts_p.java, htroot/ConfigAppearance_p.html, htroot/ConfigAppearance_p.java, htroot/ConfigBasic.html, htroot/ConfigBasic.java, htroot/ConfigHTCache_p.html, htroot/ConfigHTCache_p.java, htroot/ConfigHeuristics_p.html, htroot/ConfigHeuristics_p.java, htroot/ConfigLanguage_p.html, htroot/ConfigLanguage_p.java, htroot/ConfigLiveSearch.html, htroot/ConfigLiveSearch.java, htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java, htroot/ConfigParser.html, htroot/ConfigParser.java, htroot/ConfigPortal.html, htroot/ConfigPortal.java, htroot/ConfigProfile_p.html, htroot/ConfigProfile_p.java, htroot/ConfigProperties_p.html, htroot/ConfigProperties_p.java, htroot/ConfigRobotsTxt_p.html, htroot/ConfigRobotsTxt_p.java, htroot/ConfigSearchBox.html, htroot/ConfigSearchBox.java, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/ConfigUpdate_p.html, htroot/ConfigUpdate_p.java, htroot/Connections_p.java, htroot/ContentAnalysis_p.java, htroot/ContentControl_p.html, htroot/ContentControl_p.java, htroot/ContentIntegrationPHPBB3_p.html, htroot/ContentIntegrationPHPBB3_p.java, htroot/CookieMonitorIncoming_p.html, htroot/CookieMonitorIncoming_p.java, htroot/CookieMonitorOutgoing_p.html, htroot/CookieMonitorOutgoing_p.java, htroot/CookieTest_p.html, htroot/CookieTest_p.java, htroot/CrawlCheck_p.html, htroot/CrawlCheck_p.java, htroot/CrawlMonitorRemoteStart.java, htroot/CrawlProfileEditor_p.html, htroot/CrawlProfileEditor_p.java, htroot/CrawlResults.html, htroot/CrawlResults.java, htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java, htroot/CrawlStartScanner_p.html, htroot/CrawlStartScanner_p.java, htroot/CrawlStartSite_p.html, htroot/CrawlStartSite_p.java, htroot/Crawler_p.html, htroot/Crawler_p.java, htroot/DemoServlet.java, htroot/DemoServletInteraction.java, htroot/DemoServletRDF.java, htroot/DictionaryLoader_p.html, htroot/DictionaryLoader_p.java, htroot/Help.java, htroot/HostBrowser.html, htroot/HostBrowser.java, htroot/HostBrowser.xml, htroot/HostBrowserAdmin_p.html, htroot/IndexControlRWIs_p.html, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/IndexControlURLs_p.xml, htroot/IndexCreateLoaderQueue_p.java, htroot/IndexCreateParserErrors_p.html, htroot/IndexCreateParserErrors_p.java, htroot/IndexCreateQueues_p.html, htroot/IndexCreateQueues_p.java, htroot/IndexDeletion_p.html, htroot/IndexDeletion_p.java, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/IndexImportMediawiki_p.java, htroot/IndexImportOAIPMHList_p.java, htroot/IndexImportOAIPMH_p.java, htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, htroot/IndexSchema_p.html, htroot/IndexSchema_p.java, htroot/IndexShare_p.java, htroot/Load_MediawikiWiki.java, htroot/Load_PHPBB3.java, htroot/Load_RSS_p.html, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Messages_p.java, htroot/Network.html, htroot/Network.java, htroot/NetworkPicture.java, htroot/News.java, htroot/PeerLoadPicture.java, htroot/PerformanceConcurrency_p.html, htroot/PerformanceConcurrency_p.java, htroot/PerformanceGraph.java, htroot/PerformanceMemory_p.html, htroot/PerformanceMemory_p.java, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, htroot/PerformanceSearch_p.html, htroot/PerformanceSearch_p.java, htroot/Performance_p.java, htroot/ProxyIndexingMonitor_p.java, htroot/QuickCrawlLink_p.html, htroot/QuickCrawlLink_p.java, htroot/RankingRWI_p.html, htroot/RankingRWI_p.java, htroot/RankingSolr_p.html, htroot/RankingSolr_p.java, htroot/RegexTest.html, htroot/RegexTest.java, htroot/RemoteCrawl_p.html, htroot/RemoteCrawl_p.java, htroot/SearchEventPicture.java, htroot/ServerScannerList.java, htroot/SettingsAck_p.java, htroot/Settings_Proxy.inc, htroot/Settings_p.java, htroot/Status.html, htroot/Status.java, htroot/Status_p.inc, htroot/Steering.html, htroot/Steering.java, htroot/Supporter.java, htroot/Surftips.html, htroot/Surftips.java, htroot/Table_API_p.html, htroot/Table_API_p.java, htroot/Table_RobotsTxt_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.html, htroot/Tables_p.java, htroot/Threaddump_p.java, htroot/Trails.java, htroot/Triple_p.java, htroot/Triplestore_p.java, htroot/User.java, htroot/ViewFile.html, htroot/ViewFile.java, htroot/ViewImage.java, htroot/ViewLog_p.html, htroot/ViewLog_p.java, htroot/ViewProfile.html, htroot/ViewProfile.java, htroot/Vocabulary_p.html, htroot/Vocabulary_p.java, htroot/WatchWebStructure_p.html, htroot/WatchWebStructure_p.java, htroot/WebStructurePicture_p.java, htroot/Wiki.java, htroot/WikiHelp.html, htroot/WikiHelp.java, htroot/YMarks.html, htroot/YMarks.java, htroot/YMarks.rdf, htroot/YaCySearchPluginFF.html, htroot/YaCySearchPluginFF.java, htroot/api/blacklists.java, htroot/api/blacklists_p.java, htroot/api/bookmarks/get_bookmarks.java, htroot/api/bookmarks/get_folders.java, htroot/api/bookmarks/posts/add_p.java, htroot/api/bookmarks/posts/all.java, htroot/api/bookmarks/posts/delete_p.java, htroot/api/bookmarks/posts/get.java, htroot/api/bookmarks/tags/addTag_p.java, htroot/api/bookmarks/tags/editTag_p.java, htroot/api/bookmarks/tags/getTag.java, htroot/api/bookmarks/xbel/xbel.java, htroot/api/citation.html, htroot/api/citation.java, htroot/api/config_p.java, htroot/api/feed.java, htroot/api/feed.rss, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/latency_p.java, htroot/api/schema.java, htroot/api/schema.xml, htroot/api/status_p.java, htroot/api/status_p.xml, htroot/api/table_p.html, htroot/api/table_p.java, htroot/api/termlist_p.java, htroot/api/timeline.java, htroot/api/trail_p.java, htroot/api/version.java, htroot/api/webstructure.java, htroot/api/yacydoc.html, htroot/api/yacydoc.java, htroot/api/yacydoc.xml, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/delete_ymark.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_tags.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/get_xbel.java, htroot/api/ymarks/get_ymark.java, htroot/api/ymarks/import_ymark.java, htroot/api/ymarks/manage_tags.java, htroot/api/ymarks/test_import.html, htroot/api/ynetSearch.java, htroot/autoconfig.java, htroot/compare_yacy.java, htroot/cytag.java, htroot/env/base.css, htroot/env/grafics/JuniorMentee.gif, htroot/env/grafics/SeniorMentor.gif, htroot/env/grafics/burn-e.gif, htroot/env/grafics/construction.gif, htroot/env/grafics/external.png, htroot/env/grafics/good.png, htroot/env/grafics/link.gif, htroot/env/grafics/minitree.png, htroot/env/grafics/nok.png, htroot/env/grafics/ok.png, htroot/env/grafics/searchmode_p2p_activated_32.png, htroot/env/grafics/searchmode_p2p_deactivated_32.png, htroot/env/grafics/searchmode_stealth_activated_32.png, htroot/env/grafics/searchmode_stealth_deactivated_32.png, htroot/env/style.java, htroot/env/templates/header.template, htroot/env/templates/jqueryheader.template, htroot/env/templates/simpleheader.template, htroot/env/templates/submenuAccessTracker.template, htroot/env/templates/submenuBlacklist.template, htroot/env/templates/submenuComputation.template, htroot/env/templates/submenuConfig.template, htroot/env/templates/submenuCrawlMonitor.template, htroot/env/templates/submenuIndexControl.template, htroot/env/templates/submenuSearchConfiguration.template, htroot/env/templates/submenuSearchIntegration.template, htroot/env/templates/submenuTargetAnalysis.template, htroot/env/terminal.css, htroot/favicon.bmp, htroot/favicon.ico, htroot/favicon.png, htroot/gsa/searchresult.java, htroot/imagetest.java, htroot/index.html, htroot/index.java, htroot/interaction/GetRDF.java, htroot/interaction/PutRDF.java, htroot/interaction/Table.java, htroot/interaction/Triple.java, htroot/interaction_elements/Document_part.html, htroot/interaction_elements/Document_part.java, htroot/interaction_elements/Footer.java, htroot/interaction_elements/Loginstatus_part.java, htroot/interaction_elements/OverlayInteraction.html, htroot/interaction_elements/OverlayInteraction.java, htroot/interaction_elements/Tag_part.html, htroot/interaction_elements/Tag_part.java, htroot/js/Crawler.js, htroot/js/highslide/graphics/controlbar2.gif, htroot/js/highslide/graphics/controlbar3.gif, htroot/js/highslide/graphics/controlbar4-hover.gif, htroot/js/highslide/graphics/controlbar4.gif, htroot/js/highslide/graphics/fullexpand.gif, htroot/js/highslide/graphics/geckodimmer.png, htroot/js/highslide/graphics/loader.gif, htroot/js/highslide/graphics/loader.white.gif, htroot/js/highslide/graphics/outlines/Outlines.psd, htroot/js/highslide/graphics/outlines/beveled.png, htroot/js/highslide/graphics/outlines/drop-shadow.png, htroot/js/highslide/graphics/outlines/glossy-dark.png, htroot/js/highslide/graphics/outlines/outer-glow.png, htroot/js/highslide/graphics/outlines/rounded-black.png, htroot/js/highslide/graphics/outlines/rounded-white.png, htroot/js/highslide/graphics/resize.gif, htroot/js/highslide/graphics/zoomin.cur, htroot/js/highslide/graphics/zoomout.cur, htroot/js/highslide/highslide.js, htroot/js/query.js, htroot/js/rss2.js, htroot/js/yacy-ymarks-bookmark-actions.js, htroot/js/yacy-ymarks.js, htroot/js/yacyinteractive.js, htroot/js/yacysearch.js, htroot/mediawiki_p.java, htroot/migrateurldb_p.html, htroot/migrateurldb_p.java, htroot/opensearchdescription.java, htroot/opensearchdescription.xml, htroot/osm.java, htroot/portalsearch/yacy-portalsearch.js, htroot/processing/domaingraph/applet/domaingraph.jar, htroot/processing/domaingraph/applet/domaingraph.java, htroot/processing/domaingraph/applet/domaingraph.pde, htroot/processing/domaingraph/applet/loading.gif, htroot/processing/domaingraph/domaingraph.pde, htroot/processing/put_in_libraries/animation.zip, htroot/processing/put_in_libraries/physics.zip, htroot/rct_p.java, htroot/robots.java, htroot/robots.txt, htroot/rssTerminal.html, htroot/sharedBlacklist_p.java, htroot/solr/select.java, htroot/solr/update.java, htroot/ssitestservlet.java, htroot/suggest.java, htroot/test.java, htroot/www/welcome.html, htroot/www/welcome.java, htroot/yacy/crawlReceipt.java, htroot/yacy/hello.java, htroot/yacy/idx.java, htroot/yacy/list.java, htroot/yacy/message.java, htroot/yacy/profile.java, htroot/yacy/query.java, htroot/yacy/search.html, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacy/ui/colorpicker_images/Thumbs.db, htroot/yacy/ui/colorpicker_images/blank.gif, htroot/yacy/ui/colorpicker_images/colorpicker_background.png, htroot/yacy/ui/colorpicker_images/colorpicker_hex.png, htroot/yacy/ui/colorpicker_images/colorpicker_hsb_b.png, htroot/yacy/ui/colorpicker_images/colorpicker_hsb_h.png, htroot/yacy/ui/colorpicker_images/colorpicker_hsb_s.png, htroot/yacy/ui/colorpicker_images/colorpicker_indic.gif, htroot/yacy/ui/colorpicker_images/colorpicker_overlay.png, htroot/yacy/ui/colorpicker_images/colorpicker_rgb_b.png, htroot/yacy/ui/colorpicker_images/colorpicker_rgb_g.png, htroot/yacy/ui/colorpicker_images/colorpicker_rgb_r.png, htroot/yacy/ui/colorpicker_images/colorpicker_select.gif, htroot/yacy/ui/colorpicker_images/colorpicker_submit.png, htroot/yacy/ui/colorpicker_images/custom_background.png, htroot/yacy/ui/colorpicker_images/custom_hex.png, htroot/yacy/ui/colorpicker_images/custom_hsb_b.png, htroot/yacy/ui/colorpicker_images/custom_hsb_h.png, htroot/yacy/ui/colorpicker_images/custom_hsb_s.png, htroot/yacy/ui/colorpicker_images/custom_indic.gif, htroot/yacy/ui/colorpicker_images/custom_rgb_b.png, htroot/yacy/ui/colorpicker_images/custom_rgb_g.png, htroot/yacy/ui/colorpicker_images/custom_rgb_r.png, htroot/yacy/ui/colorpicker_images/custom_submit.png, htroot/yacy/ui/colorpicker_images/select.png, htroot/yacy/ui/colorpicker_images/select2.png, htroot/yacy/ui/colorpicker_images/slider.png, htroot/yacy/ui/css/autocomplete.css, htroot/yacy/ui/css/colorpicker.css, htroot/yacy/ui/js/jquery-1.3.2.min.js, htroot/yacy/ui/js/jquery-ui-1.7.2.min.js, htroot/yacy/ui/js/jquery.tagcloud.min.js, htroot/yacy/ui/js/jquery.tinysort.min.js, htroot/yacy/ui/yacyui-welcome.html, htroot/yacy/urls.java, htroot/yacyinteractive.html, htroot/yacyinteractive.java, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearch.rss, htroot/yacysearch_location.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, lib/commons-codec-1.7.License, lib/commons-codec-1.7.jar, lib/dependencies.txt, lib/fontbox-1.8.2.License, lib/fontbox-1.8.2.jar, lib/guava-13.0.1.jar, lib/httpclient-4.2.5.License, lib/httpclient-4.2.5.jar, lib/httpcore-4.2.4.License, lib/httpcore-4.2.4.jar, lib/httpmime-4.2.5.License, lib/httpmime-4.2.5.jar, lib/jaudiotagger-2.0.4-20111207.115108-15.License, lib/jaudiotagger-2.0.4-20111207.115108-15.jar, lib/jcifs-1.3.17-LICENSE.txt, lib/jcifs-1.3.17.jar, lib/jcl-over-slf4j-1.7.2.jar, lib/jempbox-1.8.2.License, lib/jempbox-1.8.2.jar, lib/jsoup-1.6.3.jar, lib/log4j-1.2.17.License, lib/log4j-over-slf4j-1.7.2.jar, lib/lucene-analyzers-common-4.4.0.jar, lib/lucene-analyzers-phonetic-4.4.0.jar, lib/lucene-classification-4.4.0.jar, lib/lucene-codecs-4.4.0.jar, lib/lucene-core-4.4.0.jar, lib/lucene-facet-4.4.0.jar, lib/lucene-grouping-4.4.0.jar, lib/lucene-highlighter-4.4.0.jar, lib/lucene-join-4.4.0.jar, lib/lucene-memory-4.4.0.jar, lib/lucene-misc-4.4.0.jar, lib/lucene-queries-4.4.0.jar, lib/lucene-queryparser-4.4.0.jar, lib/lucene-spatial-4.4.0.jar, lib/lucene-suggest-4.4.0.jar, lib/metadata-extractor-2.6.2.License, lib/metadata-extractor-2.6.2.jar, lib/noggit-0.5.jar, lib/pdfbox-1.8.2.License, lib/pdfbox-1.8.2.jar, lib/slf4j-api-1.7.2.jar, lib/slf4j-jdk14-1.7.2.jar, lib/solr-core-4.4.0.jar, lib/solr-solrj-4.4.0.License, lib/solr-solrj-4.4.0.jar, lib/solr.License, lib/spatial4j-0.3.jar, lib/wstx-asl-3.2.9.jar, lib/zookeeper-3.4.5.jar, libbuild/GitRevTask/GitRevTask.java, libbuild/pom.xml, libbuild/svnRevNr/svnRevNrParser.java, libt/jetty-6.1.26-patched-JETTY-1340.jar, libt/jetty-LICENSE-ASL.txt, libt/jetty-util-6.1.26-patched-JETTY-1340.jar, libt/jetty-util-LICENSE-ASL.txt, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/hi.lng, locales/ru.lng, locales/uk.lng, nbproject/project.xml, pom.xml, reconfigureYACY.sh, skins/generic_pd.css, skins/pdblue.css, source/net/yacy/YaCySearchClient.java, source/net/yacy/contentcontrol/ContentControlFilterUpdateThread.java, source/net/yacy/contentcontrol/SMWListImporter.java, source/net/yacy/contentcontrol/SMWListImporterFormatObsolete.java, source/net/yacy/contentcontrol/SMWListRow.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/ai/example/ConnectFour.java, source/net/yacy/cora/ai/example/Hanoi.java, source/net/yacy/cora/ai/example/SchwarzerPeter.java, source/net/yacy/cora/ai/example/testorder.java, source/net/yacy/cora/ai/greedy/AbstractFinding.java, source/net/yacy/cora/ai/greedy/AbstractModel.java, source/net/yacy/cora/ai/greedy/Agent.java, source/net/yacy/cora/ai/greedy/Asset.java, source/net/yacy/cora/ai/greedy/Attempts.java, source/net/yacy/cora/ai/greedy/Battle.java, source/net/yacy/cora/ai/greedy/Challenge.java, source/net/yacy/cora/ai/greedy/Context.java, source/net/yacy/cora/ai/greedy/ContextFactory.java, source/net/yacy/cora/ai/greedy/Engine.java, source/net/yacy/cora/ai/greedy/Finding.java, source/net/yacy/cora/ai/greedy/Goal.java, source/net/yacy/cora/ai/greedy/Model.java, source/net/yacy/cora/ai/greedy/Role.java, source/net/yacy/cora/ai/greedy/Unirole.java, source/net/yacy/cora/date/GenericFormatter.java, source/net/yacy/cora/date/ISO8601Formatter.java, source/net/yacy/cora/date/MicroDate.java, source/net/yacy/cora/document/ASCII.java, source/net/yacy/cora/document/Hit.java, source/net/yacy/cora/document/JSONArray.java, source/net/yacy/cora/document/JSONObject.java, source/net/yacy/cora/document/JSONTokener.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/document/RSSFeed.java, source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/document/RSSReader.java, source/net/yacy/cora/document/UTF8.java, source/net/yacy/cora/document/WordCache.java, source/net/yacy/cora/document/analysis/Classification.java, source/net/yacy/cora/document/analysis/EnhancedTextProfileSignature.java, source/net/yacy/cora/federate/SearchAccumulator.java, source/net/yacy/cora/federate/opensearch/OpenSearchConnector.java, source/net/yacy/cora/federate/opensearch/SRURSSConnector.java, source/net/yacy/cora/federate/solr/FailType.java, source/net/yacy/cora/federate/solr/ProcessType.java, source/net/yacy/cora/federate/solr/Ranking.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/SchemaDeclaration.java, source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/cora/federate/solr/SolrType.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ShardSelection.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/federate/solr/instance/ResponseAccumulator.java, source/net/yacy/cora/federate/solr/instance/ServerMirror.java, source/net/yacy/cora/federate/solr/instance/ServerShard.java, source/net/yacy/cora/federate/solr/instance/ShardInstance.java, source/net/yacy/cora/federate/solr/instance/SolrInstance.java, source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/cora/federate/yacy/CacheStrategy.java, source/net/yacy/cora/federate/yacy/Distribution.java, source/net/yacy/cora/federate/yacy/Peer.java, source/net/yacy/cora/federate/yacy/Peers.java, source/net/yacy/cora/federate/yacy/api/Network.java, source/net/yacy/cora/geo/GeoLocation.java, source/net/yacy/cora/geo/GeoPoint.java, source/net/yacy/cora/geo/GeonamesLocation.java, source/net/yacy/cora/geo/IntegerGeoPoint.java, source/net/yacy/cora/geo/Locations.java, source/net/yacy/cora/geo/OpenGeoDBLocation.java, source/net/yacy/cora/geo/OverarchingLocation.java, source/net/yacy/cora/language/phonetic/DoubleMetaphone.java, source/net/yacy/cora/language/phonetic/Metaphone.java, source/net/yacy/cora/language/phonetic/Phonetic.java, source/net/yacy/cora/language/phonetic/Soundex.java, source/net/yacy/cora/language/synonyms/AutotaggingLibrary.java, source/net/yacy/cora/language/synonyms/SynonymLibrary.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/Vocabulary.java, source/net/yacy/cora/lod/vocabulary/AnnoteaA.java, source/net/yacy/cora/lod/vocabulary/AnnoteaB.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/DCElements.java, source/net/yacy/cora/lod/vocabulary/DCTerms.java, source/net/yacy/cora/lod/vocabulary/DMOZ.java, source/net/yacy/cora/lod/vocabulary/DublinCore.java, source/net/yacy/cora/lod/vocabulary/Foaf.java, source/net/yacy/cora/lod/vocabulary/Geo.java, source/net/yacy/cora/lod/vocabulary/HttpHeader.java, source/net/yacy/cora/lod/vocabulary/Owl.java, source/net/yacy/cora/lod/vocabulary/Rdf.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/lod/vocabulary/YaCyMetadata.java, source/net/yacy/cora/order/Base64Order.java, source/net/yacy/cora/order/CloneableIterator.java, source/net/yacy/cora/order/CloneableMapIterator.java, source/net/yacy/cora/order/Digest.java, source/net/yacy/cora/order/NaturalOrder.java, source/net/yacy/cora/order/StringOrder.java, source/net/yacy/cora/protocol/ByteArrayBody.java, source/net/yacy/cora/protocol/ClientIdentification.java, source/net/yacy/cora/protocol/ConnectionInfo.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/TimeoutRequest.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/LinkExtractor.java, source/net/yacy/cora/protocol/http/ProxySettings.java, source/net/yacy/cora/sorting/AbstractScoreMap.java, source/net/yacy/cora/sorting/ClusteredScoreMap.java, source/net/yacy/cora/sorting/OrderedScoreMap.java, source/net/yacy/cora/sorting/ScoreMap.java, source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java, source/net/yacy/cora/storage/ConcurrentARC.java, source/net/yacy/cora/storage/Configuration.java, source/net/yacy/cora/storage/Files.java, source/net/yacy/cora/storage/HandleMap.java, source/net/yacy/cora/storage/HandleSet.java, source/net/yacy/cora/storage/KeyList.java, source/net/yacy/cora/storage/SimpleARC.java, source/net/yacy/cora/storage/SizeLimitedMap.java, source/net/yacy/cora/storage/SizeLimitedSet.java, source/net/yacy/cora/storage/ZIPReader.java, source/net/yacy/cora/storage/ZIPWriter.java, source/net/yacy/cora/util/ByteArray.java, source/net/yacy/cora/util/ByteBuffer.java, source/net/yacy/cora/util/CommonPattern.java, source/net/yacy/cora/util/ConcurrentLog.java, source/net/yacy/cora/util/LookAheadIterator.java, source/net/yacy/cora/util/Memory.java, source/net/yacy/cora/util/SpaceExceededException.java, source/net/yacy/cora/util/StringBuilderComparator.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlQueue.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/HarvestProcess.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/crawler/data/NoticedURL.java, source/net/yacy/crawler/data/ResultImages.java, source/net/yacy/crawler/data/ResultURLs.java, source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/ImporterException.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/crawler/retrieval/Request.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/crawler/retrieval/SitemapImporter.java, source/net/yacy/crawler/retrieval/URLRewriterLibrary.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/crawler/robots/RobotsTxtEntry.java, source/net/yacy/crawler/robots/RobotsTxtParser.java, source/net/yacy/data/BlogBoard.java, source/net/yacy/data/BlogBoardComments.java, source/net/yacy/data/BookmarkDate.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/data/BookmarksDB.java, source/net/yacy/data/DidYouMean.java, source/net/yacy/data/Diff.java, source/net/yacy/data/ListManager.java, source/net/yacy/data/MessageBoard.java, source/net/yacy/data/Translator.java, source/net/yacy/data/URLLicense.java, source/net/yacy/data/UserDB.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/list/ListAccumulator.java, source/net/yacy/data/list/XMLBlacklistImporter.java, source/net/yacy/data/wiki/AbstractWikiParser.java, source/net/yacy/data/wiki/WikiBoard.java, source/net/yacy/data/wiki/WikiCode.java, source/net/yacy/data/wiki/WikiParser.java, source/net/yacy/data/ymark/MonitoredReader.java, source/net/yacy/data/ymark/TablesRowComparator.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/data/ymark/YMarkDMOZImporter.java, source/net/yacy/data/ymark/YMarkDate.java, source/net/yacy/data/ymark/YMarkEntry.java, source/net/yacy/data/ymark/YMarkHTMLImporter.java, source/net/yacy/data/ymark/YMarkImporter.java, source/net/yacy/data/ymark/YMarkJSONImporter.java, source/net/yacy/data/ymark/YMarkMetadata.java, source/net/yacy/data/ymark/YMarkRDF.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/data/ymark/YMarkTag.java, source/net/yacy/data/ymark/YMarkUtil.java, source/net/yacy/data/ymark/YMarkXBELImporter.java, source/net/yacy/dbtest.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/ImageParser.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/Parser.java, source/net/yacy/document/SentenceReader.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/WordTokenizer.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/SurrogateReader.java, source/net/yacy/document/content/dao/DatabaseConnection.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/language/LanguageStatistics.java, source/net/yacy/document/language/LanguageStatisticsHolder.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/AbstractScraper.java, source/net/yacy/document/parser/html/CharacterCoding.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ContentTransformer.java, source/net/yacy/document/parser/html/EmbedEntry.java, source/net/yacy/document/parser/html/Evaluation.java, source/net/yacy/document/parser/html/ImageEntry.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/bmpParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/images/icoParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaTripleImpl.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/document/parser/xml/ODContentHandler.java, source/net/yacy/document/parser/xml/opensearchdescriptionReader.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/gui/InfoPage.java, source/net/yacy/gui/YaCyApp.java, source/net/yacy/gui/framework/Application.java, source/net/yacy/gui/framework/Browser.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BEncodedHeapBag.java, source/net/yacy/kelondro/blob/BEncodedHeapShard.java, source/net/yacy/kelondro/blob/BLOB.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/Gap.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapColumnIndex.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Stack.java, source/net/yacy/kelondro/blob/Stacks.java, source/net/yacy/kelondro/blob/TableColumnIndexException.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/blob/TablesColumnBLOBIndex.java, source/net/yacy/kelondro/blob/TablesColumnIndex.java, source/net/yacy/kelondro/blob/TablesColumnRAMIndex.java, source/net/yacy/kelondro/data/citation/CitationReference.java, source/net/yacy/kelondro/data/citation/CitationReferenceFactory.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/navigation/NavigationReferenceRow.java, source/net/yacy/kelondro/data/navigation/NavigationReferenceVars.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/data/word/WordReference.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/BinSearch.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/Cache.java, source/net/yacy/kelondro/index/Column.java, source/net/yacy/kelondro/index/Index.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/RAMIndex.java, source/net/yacy/kelondro/index/RAMIndexCluster.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/ByteCountInputStream.java, source/net/yacy/kelondro/io/CachedFileReader.java, source/net/yacy/kelondro/io/CachedFileWriter.java, source/net/yacy/kelondro/io/CachedRecords.java, source/net/yacy/kelondro/io/CharBuffer.java, source/net/yacy/kelondro/io/RandomAccessIO.java, source/net/yacy/kelondro/io/Records.java, source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java, source/net/yacy/kelondro/logging/ConsoleOutHandler.java, source/net/yacy/kelondro/logging/GuiHandler.java, source/net/yacy/kelondro/logging/LogParser.java, source/net/yacy/kelondro/logging/LogalizerHandler.java, source/net/yacy/kelondro/logging/SimpleLogFormatter.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/kelondro/rwi/AbstractBufferedIndex.java, source/net/yacy/kelondro/rwi/AbstractIndex.java, source/net/yacy/kelondro/rwi/AbstractReference.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/Index.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/rwi/IndexReader.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/rwi/ReferenceFactory.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/rwi/TermSearch.java, source/net/yacy/kelondro/table/ChunkIterator.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SQLTable.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/BDecoder.java, source/net/yacy/kelondro/util/BEncoder.java, source/net/yacy/kelondro/util/Bitfield.java, source/net/yacy/kelondro/util/ConsoleInterface.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/Formatter.java, source/net/yacy/kelondro/util/GenerationMemoryStrategy.java, source/net/yacy/kelondro/util/ISO639.java, source/net/yacy/kelondro/util/MemoryControl.java, source/net/yacy/kelondro/util/MemoryStrategy.java, source/net/yacy/kelondro/util/MergeIterator.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/kelondro/util/ReverseMapIterator.java, source/net/yacy/kelondro/util/RotateIterator.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/kelondro/util/StackIterator.java, source/net/yacy/kelondro/util/StandardMemoryStrategy.java, source/net/yacy/kelondro/workflow/AbstractBlockingThread.java, source/net/yacy/kelondro/workflow/AbstractBusyThread.java, source/net/yacy/kelondro/workflow/AbstractThread.java, source/net/yacy/kelondro/workflow/BusyThread.java, source/net/yacy/kelondro/workflow/InstantBlockingThread.java, source/net/yacy/kelondro/workflow/InstantBusyThread.java, source/net/yacy/kelondro/workflow/WorkflowJob.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/migration.java, source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/Transmission.java, source/net/yacy/peers/graphics/Banner.java, source/net/yacy/peers/graphics/BannerData.java, source/net/yacy/peers/graphics/EncodedImage.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/graphics/ProfilingGraph.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/peers/operation/yacySeedUploadFile.java, source/net/yacy/peers/operation/yacySeedUploadFtp.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/peers/operation/yacySeedUploader.java, source/net/yacy/peers/operation/yacyVersion.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/repository/RegexHelper.java, source/net/yacy/search/EventTracker.java, source/net/yacy/search/IndexingQueueEntry.java, source/net/yacy/search/MemoryTracker.java, source/net/yacy/search/ResourceObserver.java, source/net/yacy/search/Shutdown.java, source/net/yacy/search/StorageQueueEntry.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/ReindexSolrBusyThread.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/HeuristicResult.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SearchEventCache.java, source/net/yacy/search/query/SearchEventType.java, source/net/yacy/search/query/SecondarySearchSuperviser.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/RankingProfile.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/server/http/AlternativeDomainNames.java, source/net/yacy/server/http/AugmentedHtmlStream.java, source/net/yacy/server/http/ChunkedInputStream.java, source/net/yacy/server/http/ChunkedOutputStream.java, source/net/yacy/server/http/ContentLengthInputStream.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/http/MultiOutputStream.java, source/net/yacy/server/http/ProxyLogFormatter.java, source/net/yacy/server/http/RobotsTxtConfig.java, source/net/yacy/server/http/ServerSideIncludes.java, source/net/yacy/server/http/TemplateEngine.java, source/net/yacy/server/serverAccessTracker.java, source/net/yacy/server/serverClassLoader.java, source/net/yacy/server/serverCore.java, source/net/yacy/server/serverCoreSocket.java, source/net/yacy/server/serverHandler.java, source/net/yacy/server/serverObjects.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/server/serverSwitchAbstractAction.java, source/net/yacy/server/servletProperties.java, source/net/yacy/upnp/Discovery.java, source/net/yacy/upnp/DiscoveryAdvertisement.java, source/net/yacy/upnp/DiscoveryListener.java, source/net/yacy/upnp/HttpResponse.java, source/net/yacy/upnp/JXPathParser.java, source/net/yacy/upnp/ServicesEventing.java, source/net/yacy/upnp/devices/UPNPRootDevice.java, source/net/yacy/upnp/impls/InternetGatewayDevice.java, source/net/yacy/upnp/messages/ActionMessage.java, source/net/yacy/upnp/messages/ActionMessageResponseParser.java, source/net/yacy/upnp/messages/StateVariableMessage.java, source/net/yacy/upnp/messages/StateVariableResponseParser.java, source/net/yacy/upnp/services/ISO8601Date.java, source/net/yacy/upnp/services/UPNPService.java, source/net/yacy/utils/CryptoLib.java, source/net/yacy/utils/ListDirs.java, source/net/yacy/utils/PKCS12Tool.java, source/net/yacy/utils/SignatureOutputStream.java, source/net/yacy/utils/UPnP.java, source/net/yacy/utils/bitfield.java, source/net/yacy/utils/crypt.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/utils/disorderHeap.java, source/net/yacy/utils/disorderSet.java, source/net/yacy/utils/enumerateFiles.java, source/net/yacy/utils/gzip.java, source/net/yacy/utils/loaderCore.java, source/net/yacy/utils/loaderProcess.java, source/net/yacy/utils/loaderThreads.java, source/net/yacy/utils/nxTools.java, source/net/yacy/utils/tarTools.java, source/net/yacy/utils/whois.java, source/net/yacy/visualization/AnimationGIF.java, source/net/yacy/visualization/Captcha.java, source/net/yacy/visualization/ChartPlotter.java, source/net/yacy/visualization/CircleTool.java, source/net/yacy/visualization/GraphPlotter.java, source/net/yacy/visualization/HexGridPlotter.java, source/net/yacy/visualization/PrintTool.java, source/net/yacy/visualization/RasterPlotter.java, source/net/yacy/yacy.java, source/org/apache/tools/tar/TarInputStream.java, startYACY.bat, startYACY.sh, stopYACY.bat, stopYACY.sh, test/de/anomic/document/ParserTest.java, test/de/anomic/yacy/yacyURLTest.java, test/de/anomic/yacy/yacyVersionTest.java, test/net/yacy/cora/document/MultiProtocolURITest.java, test/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnectorTest.java, yacy.stopwords |
Wed Sep 04 23:11:53 CEST 2013 by Michael Peter Christen | added two new fields, exact_signature_copycount_i and fuzzy_signature_copycount_i, which count the number of copies of non-unique documents and assigns this to each document. Thus, each document there is a number assigned which shows how many copies of this document exists. These fields are disabled by default. Changed Files: defaults/solr.collection.schema, source/net/yacy/cora/federate/solr/ProcessType.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java |
Mon Sep 02 18:55:38 CEST 2013 by Michael Peter Christen | redesign of the image search process (with much better results, unfortunately the index schema has changed and p2p image search will not be muchmuch better until many people update) Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, htroot/yacysearchitem.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/snippet/ResultEntry.java |
Sun Aug 25 00:13:48 CEST 2013 by Michael Peter Christen | added a feature to set a collection for a crawl result based on a regular expression on th url: the collection attribut for a crawl start may be now either a token or a list of tokens, seperated by ',' where a token is either a string or a pair <string,pattern> where the string is separated to the pattern with a ':' and the string is assigned to the document as collection only if the pattern matches with the url. Changed Files: htroot/Load_RSS_p.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Thu Aug 22 14:23:47 CEST 2013 by Michael Peter Christen | Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent. Changed Files: defaults/yacy.init, htroot/Bookmarks.java, htroot/ConfigAppearance_p.java, htroot/ConfigLanguage_p.java, htroot/CrawlCheck_p.java, htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexImportOAIPMHList_p.java, htroot/IndexImportOAIPMH_p.java, htroot/Load_RSS_p.java, htroot/Network.java, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/QuickCrawlLink_p.java, htroot/ViewFile.java, htroot/ViewImage.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/latency_p.java, htroot/api/webstructure.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/import_ymark.java, htroot/sharedBlacklist_p.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/federate/opensearch/SRURSSConnector.java, source/net/yacy/cora/federate/yacy/api/Network.java, source/net/yacy/cora/protocol/ClientIdentification.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/CrawlQueue.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/crawler/data/NoticedURL.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/crawler/retrieval/SitemapImporter.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/crawler/robots/RobotsTxtEntry.java, source/net/yacy/crawler/robots/RobotsTxtParser.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/data/ymark/YMarkMetadata.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/xml/opensearchdescriptionReader.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/utils/loaderThreads.java, source/net/yacy/yacy.java |
Tue Aug 20 15:46:04 CEST 2013 by Michael Peter Christen | - refactoring - generalized writing of url attributes to solr documents - added more url attributes to error documents Changed Files: htroot/IndexControlRWIs_p.java, htroot/Wiki.java, htroot/yacy/search.java, source/net/yacy/crawler/robots/RobotsTxtEntry.java, source/net/yacy/dbtest.java, source/net/yacy/document/Document.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/MapColumnIndex.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/blob/TablesColumnBLOBIndex.java, source/net/yacy/kelondro/data/citation/CitationReference.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/graphics/EncodedImage.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/server/http/ChunkedOutputStream.java, source/net/yacy/server/http/HTTPDFileHandler.java, source/net/yacy/server/http/HTTPDemon.java, source/net/yacy/server/http/ServerSideIncludes.java, source/net/yacy/server/http/TemplateEngine.java, source/net/yacy/server/serverCore.java, source/net/yacy/visualization/RasterPlotter.java |
Wed Jul 31 10:49:26 CEST 2013 by Michael Peter Christen | fix for default boolean post values: the default value MUST NOT be TRUE, because it's normal that a boolean value is missing in the post argument if a checkbox is not selected. Added also some style enhancements to IndexFederated, removed the Solr attachment manual and replaced it with a link to the wiki which explains this in more detail. Changed Files: htroot/AccessPicture_p.java, htroot/ConfigBasic.java, htroot/ConfigPortal.java, htroot/ConfigSearchPage_p.java, htroot/ConfigUpdate_p.java, htroot/Connections_p.java, htroot/Crawler_p.java, htroot/HostBrowser.java, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/NetworkPicture.java, htroot/PeerLoadPicture.java, htroot/Status.java, htroot/Table_API_p.java, htroot/Threaddump_p.java, htroot/ViewFile.java, htroot/api/ymarks/import_ymark.java, htroot/opensearchdescription.java, source/net/yacy/server/serverObjects.java |
Tue Jul 30 14:01:16 CEST 2013 by Michael Peter Christen | migration to solr 4.4.0 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/jcifs-1.3.17-LICENSE.txt, lib/jcifs-1.3.17.jar, lib/lucene-analyzers-common-4.4.0.jar, lib/lucene-analyzers-phonetic-4.4.0.jar, lib/lucene-classification-4.4.0.jar, lib/lucene-codecs-4.4.0.jar, lib/lucene-core-4.4.0.jar, lib/lucene-facet-4.4.0.jar, lib/lucene-grouping-4.4.0.jar, lib/lucene-highlighter-4.4.0.jar, lib/lucene-join-4.4.0.jar, lib/lucene-memory-4.4.0.jar, lib/lucene-misc-4.4.0.jar, lib/lucene-queries-4.4.0.jar, lib/lucene-queryparser-4.4.0.jar, lib/lucene-spatial-4.4.0.jar, lib/lucene-suggest-4.4.0.jar, lib/solr-core-4.4.0.jar, lib/solr-solrj-4.4.0.License, lib/solr-solrj-4.4.0.jar, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java |
Tue Jul 30 12:48:57 CEST 2013 by Michael Peter Christen | replaced the single-text description solr field with a multi-value description_txt text field Changed Files: defaults/solr.collection.schema, htroot/Load_RSS_p.java, htroot/api/feed.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/gsa/searchresult.java, htroot/solr/select.java, htroot/yacysearch_location.java, source/net/yacy/cora/document/Hit.java, source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/document/analysis/EnhancedTextProfileSignature.java, source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/data/ymark/YMarkAutoTagger.java, source/net/yacy/data/ymark/YMarkEntry.java, source/net/yacy/data/ymark/YMarkMetadata.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/SurrogateReader.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/server/http/HTTPDFileHandler.java |
Commit | Description |
---|---|
Sun Feb 09 15:04:46 CET 2014 by Michael Peter Christen | catch solr close exceptions Changed Files: source/net/yacy/search/index/Fulltext.java |
Sun Feb 09 12:36:14 CET 2014 by Michael Peter Christen | enhanced memory protection and OOM exception handling in Solr connector Changed Files: source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java |
Sun Feb 09 12:25:55 CET 2014 by Michael Peter Christen | fixed setting of shortMemoryStatus in MemoryControl Changed Files: source/net/yacy/kelondro/util/GenerationMemoryStrategy.java, source/net/yacy/kelondro/util/MemoryControl.java, source/net/yacy/kelondro/util/MemoryStrategy.java, source/net/yacy/kelondro/util/StandardMemoryStrategy.java |
Mon Feb 03 13:38:26 CET 2014 by Michael Peter Christen | fixed 'resource leak' warning Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Fri Jan 31 23:03:01 CET 2014 by reger | fix vocabulary navigator checkbox selection (from last commit) Changed Files: htroot/yacysearchtrailer.java |
Mon Jan 27 15:56:02 CET 2014 by Michael Peter Christen | fix auth for forced ping Changed Files: htroot/Network.java |
Mon Jan 27 15:43:08 CET 2014 by Michael Peter Christen | fixed missing corona in network picture Changed Files: htroot/NetworkPicture.java |
Wed Jan 22 21:33:11 CET 2014 by orbiter | fix for bad-formatted lonlat Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataRow.java |
Wed Jan 22 20:59:53 CET 2014 by orbiter | fix for http://bugs.yacy.net/view.php?id=354 Changed Files: source/net/yacy/search/query/SearchEvent.java |
Wed Jan 22 20:54:03 CET 2014 by orbiter | fix for OOM during remote search and too high load protection Changed Files: source/net/yacy/peers/RemoteSearch.java |
Sun Jan 19 14:42:26 CET 2014 by Michael Peter Christen | removed debug code Changed Files: source/net/yacy/kelondro/index/RowHandleMap.java |
Sat Jan 18 00:03:44 CET 2014 by sixcooler | NP-fix, which was found on a Pi under 'havy' load Changed Files: source/net/yacy/http/AbstractRemoteHandler.java |
Fri Jan 17 01:00:02 CET 2014 by reger | fix role name in GSA servlet Changed Files: source/net/yacy/http/servlets/GSAsearchServlet.java |
Tue Jan 07 22:44:24 CET 2014 by reger | fix: NPE on shutdown via script Changed Files: source/net/yacy/yacy.java |
Tue Jan 07 20:34:37 CET 2014 by reger | fix: display proper error msg on admin user change Changed Files: htroot/ConfigAccounts_p.java |
Tue Jan 07 01:04:33 CET 2014 by sixcooler | fix double-escaped urls from proxy-usage Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/http/ProxyHandler.java |
Mon Jan 06 01:48:54 CET 2014 by orbiter | fixed shutdown authenticaton problem Changed Files: htroot/ConfigAppearance_p.java, htroot/ConfigLanguage_p.java, htroot/CrawlStartScanner_p.java, htroot/Table_API_p.java, htroot/sharedBlacklist_p.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/federate/yacy/api/Network.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/utils/loaderThreads.java, source/net/yacy/yacy.java |
Wed Jan 01 10:33:20 CET 2014 by reger | fix external link (open in new tab) Changed Files: htroot/IndexControlRWIs_p.html |
Tue Dec 31 15:25:44 CET 2013 by orbiter | added security patch Changed Files: source/net/yacy/http/servlets/YaCyProxyServlet.java |
Mon Dec 23 10:52:02 CET 2013 by Michael Peter Christen | wrong logger fix Changed Files: source/net/yacy/cora/federate/solr/Ranking.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Mon Dec 23 10:47:11 CET 2013 by Michael Peter Christen | alphabetic order fix Changed Files: addon/YaCy.app/Contents/Info.plist, build.xml |
Fri Dec 20 01:51:51 CET 2013 by Michael Peter Christen | fixed the SolrServlet Changed Files: htroot/solr/select.java, source/net/yacy/cora/federate/solr/SolrServlet.java |
Thu Dec 19 15:35:31 CET 2013 by Michael Peter Christen | next try to fix m4 Changed Files: addon/yacyInit.m4 |
Thu Dec 19 14:40:11 CET 2013 by Michael Peter Christen | fix for deb m4 Changed Files: addon/yacyInit.m4 |
Thu Dec 12 10:38:32 CET 2013 by Michael Peter Christen | fix for bad ErrorCache.exists test (bug from latest commit) Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/index/ErrorCache.java |
Fri Dec 06 14:30:08 CET 2013 by Michael Peter Christen | IPv6 fixes Changed Files: htroot/SettingsAck_p.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java |
Wed Dec 04 18:44:23 CET 2013 by Michael Peter Christen | fixed bug with image search in yacyinteractive Changed Files: htroot/js/yacyinteractive.js, htroot/yacysearchitem.java |
Wed Dec 04 13:48:01 CET 2013 by Michael Peter Christen | fixed urlmask which was not able to combine several constraints Changed Files: source/net/yacy/search/query/QueryParams.java |
Wed Dec 04 01:56:38 CET 2013 by Michael Peter Christen | fixed a NPE in surrogat processing Changed Files: source/net/yacy/search/schema/WebgraphConfiguration.java |
Wed Dec 04 01:54:14 CET 2013 by Michael Peter Christen | fixed a division by zero Changed Files: htroot/ConfigHTCache_p.java |
Fri Nov 29 23:23:47 CET 2013 by reger | fix NPE on mime detection of unknown file extension Changed Files: source/net/yacy/http/YaCyDefaultServlet.java |
Tue Nov 26 11:47:04 CET 2013 by Michael Peter Christen | include/exclude size NPE fix (recently added) Changed Files: source/net/yacy/search/query/QueryGoal.java |
Fri Nov 22 14:15:52 CET 2013 by Michael Peter Christen | fixed seedlist (hash was missing) Changed Files: htroot/yacy/seedlist.java |
Fri Nov 22 10:07:08 CET 2013 by Michael Peter Christen | fixed json Changed Files: htroot/yacy/seedlist.json |
Mon Nov 18 22:47:35 CET 2013 by orbiter | purge a lucene cache - possible memory leak fix Changed Files: htroot/IndexControlURLs_p.java, source/net/yacy/search/Switchboard.java |
Sun Nov 17 00:17:25 CET 2013 by Michael Peter Christen | fix for another XSS bug Changed Files: htroot/portalsearch/yacy-portalsearch.js |
Sat Nov 16 20:34:31 CET 2013 by orbiter | fixed some XSS bugs reported by Marius from http://ctf365.com/ Changed Files: htroot/User.java, htroot/ViewProfile.java, source/net/yacy/server/http/HTTPDFileHandler.java |
Sat Nov 16 11:33:35 CET 2013 by Michael Peter Christen | fix for wrong counter Changed Files: source/net/yacy/search/index/Segment.java |
Sat Nov 16 08:23:21 CET 2013 by Michael Peter Christen | fixed and enhanced postprocessing Changed Files: source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Sat Nov 16 08:22:29 CET 2013 by Michael Peter Christen | fixed termination bug in Solr Connector Changed Files: source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java |
Fri Nov 15 15:41:12 CET 2013 by Michael Peter Christen | enhanced and fixed postprocessing Changed Files: htroot/api/webstructure.java, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Fri Nov 15 15:39:35 CET 2013 by Michael Peter Christen | fixed several problems in solr connectors Changed Files: source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Fri Nov 15 15:37:01 CET 2013 by Michael Peter Christen | fixed keepalive min value Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java |
Fri Nov 15 15:36:24 CET 2013 by Michael Peter Christen | fixed logging for remote solr configuration Changed Files: defaults/yacy.logging |
Wed Nov 13 13:41:18 CET 2013 by Michael Peter Christen | removed debug code for existsByIds Changed Files: defaults/yacy.init, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Sat Nov 09 02:37:12 CET 2013 by Michael Peter Christen | added debug code Changed Files: defaults/yacy.init, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Wed Nov 06 19:22:26 CET 2013 by sixcooler | URL-export:add query and fix for cast-class-exception Changed Files: htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, source/net/yacy/search/index/Fulltext.java |
Mon Nov 04 10:16:20 CET 2013 by Michael Peter Christen | fix in html parser Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/document/parser/html/CharacterCoding.java, source/net/yacy/document/parser/html/ContentScraper.java |
Sat Nov 02 00:18:24 CET 2013 by reger | fix typo in GitRevTask (branch) Changed Files: libbuild/GitRevTask/GitRevTask.java |
Mon Oct 21 12:49:32 CEST 2013 by Michael Peter Christen | fixed sitemap crawl start Changed Files: htroot/Crawler_p.java |
Wed Oct 16 11:49:04 CEST 2013 by Michael Peter Christen | fixed webgraph postprocessing (did nothing and repeated to do this...) Changed Files: source/net/yacy/search/schema/WebgraphConfiguration.java |
Mon Oct 14 23:32:24 CEST 2013 by sixcooler | bump to httpclient-4.3.1 - a bugfix release Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/dependencies.txt, lib/httpclient-4.3.1.License, lib/httpclient-4.3.1.jar, lib/httpmime-4.3.1.License, lib/httpmime-4.3.1.jar, nbproject/project.xml |
Mon Oct 14 04:29:34 CEST 2013 by reger | fix last commit (not needed test of GZipInputStream) Changed Files: source/net/yacy/http/YaCyDefaultServlet.java |
Fri Oct 11 00:44:04 CEST 2013 by sixcooler | NPE fix for my last commit Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java |
Wed Oct 09 16:17:50 CEST 2013 by Michael Peter Christen | fix for didyoumean, added also more asian alphabets Changed Files: source/net/yacy/data/DidYouMean.java, source/net/yacy/search/index/Segment.java |
Wed Oct 09 15:10:03 CEST 2013 by Michael Peter Christen | enhanced ranking; patches to replace old ranking Changed Files: defaults/yacy.init, htroot/RankingSolr_p.java, source/net/yacy/search/Switchboard.java |
Wed Oct 09 11:51:07 CEST 2013 by bhoerdzn | remove debugging code intercepting post requests Changed Files: htroot/CrawlStartExpert_p.html |
Wed Oct 09 11:41:15 CEST 2013 by bhoerdzn | fix typo in parsing "cachePolicy" parameter Changed Files: htroot/CrawlStartExpert_p.java |
Wed Oct 09 11:32:17 CEST 2013 by bhoerdzn | Fixed template vars for "deleteold". Fixed parsing "deleteold" parameter. Stop "setState" overwriting "deletold" state on load. Changed Files: htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java |
Wed Oct 09 10:00:16 CEST 2013 by bhoerdzn | fixed wrong parameter check Changed Files: htroot/CrawlStartExpert_p.java |
Tue Oct 08 15:04:31 CEST 2013 by bhoerdzn | Use list template for all other option lists. Fixed some template expressions. Changed Files: htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java |
Mon Oct 07 23:51:08 CEST 2013 by Michael Peter Christen | fixed search navigation Changed Files: htroot/yacysearch.java |
Mon Oct 07 12:59:54 CEST 2013 by Michael Peter Christen | fix for rss reader, see http://bugs.yacy.net/view.php?id=294 Changed Files: source/net/yacy/cora/document/feed/RSSReader.java |
Mon Sep 30 04:35:17 CEST 2013 by reger | fix : forgot adding security handler Changed Files: source/net/yacy/http/HttpServer.java |
Mon Sep 30 02:50:53 CEST 2013 by reger | fix NPE on modified since check ( Response.requestHeader allowed to be null) Changed Files: source/net/yacy/crawler/retrieval/Response.java |
Sat Sep 28 15:45:33 CEST 2013 by orbiter | added classpath for debugging Changed Files: .classpath |
Fri Sep 27 15:09:04 CEST 2013 by Michael Peter Christen | fix for canonical-from-http-header feature Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Thu Sep 26 13:41:52 CEST 2013 by Michael Peter Christen | fix for crawl domain counter limitation (limit was reached too early) Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/search/Switchboard.java |
Mon Sep 23 03:05:09 CEST 2013 by reger | fix not necessary use of DigestURL Changed Files: source/net/yacy/http/YaCyDefaultServlet.java, source/net/yacy/http/YaCySecurityHandler.java |
Sat Sep 21 21:03:58 CEST 2013 by orbiter | fix for npe during location search Changed Files: htroot/yacysearch_location.java, source/net/yacy/peers/Protocol.java |
Sat Sep 21 10:20:13 CEST 2013 by orbiter | fix for 100%CPU problem in error cache cleaning process Changed Files: source/net/yacy/search/index/ErrorCache.java |
Sun Sep 15 23:54:46 CEST 2013 by Michael Peter Christen | fix for parsing of image links inside of anchor links (image-links) Changed Files: source/net/yacy/document/parser/html/TransformerWriter.java |
Tue Sep 10 01:42:08 CEST 2013 by reger | added read parameter from multi-part form fields (to nasty quick-fix) Changed Files: source/net/yacy/http/TemplateHandler.java |
Tue Sep 03 12:22:29 CEST 2013 by Michael Peter Christen | fixed wrong facet count Changed Files: source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Tue Sep 03 10:49:09 CEST 2013 by Michael Peter Christen | fix for content domain classification in URIMetadataNode Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java |
Tue Sep 03 07:55:21 CEST 2013 by Michael Peter Christen | fix for catchall query goal for image search Changed Files: source/net/yacy/search/query/QueryGoal.java |
Tue Sep 03 07:54:49 CEST 2013 by Michael Peter Christen | fix for dc:description field Changed Files: htroot/ViewFile.java |
Mon Sep 02 20:02:26 CEST 2013 by Michael Peter Christen | one more fix for image search Changed Files: source/net/yacy/search/query/QueryGoal.java |
Mon Sep 02 08:05:42 CEST 2013 by Michael Peter Christen | fix for solr/gsa result logging Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java |
Sun Sep 01 13:28:37 CEST 2013 by orbiter | npe fix Changed Files: htroot/solr/select.java |
Sat Aug 24 23:36:50 CEST 2013 by Michael Peter Christen | NPE during shutdown fix Changed Files: source/net/yacy/peers/Dispatcher.java |
Tue Aug 20 16:10:26 CEST 2013 by Michael Peter Christen | fix for http://bugs.yacy.net/view.php?id=286 Changed Files: source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/document/RSSReader.java |
Sun Aug 18 10:37:25 CEST 2013 by Michael Peter Christen | fix for bad snippets in gsa api Changed Files: source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java |
Sat Aug 17 09:32:24 CEST 2013 by Michael Peter Christen | html fix Changed Files: htroot/ConfigAccounts_p.html, htroot/ConfigPortal.html, htroot/DictionaryLoader_p.html, htroot/IndexFederated_p.html, htroot/IndexSchema_p.html, htroot/PerformanceSearch_p.html, htroot/yacy/ui/yacyui-welcome.html |
Wed Aug 14 20:59:46 CEST 2013 by reger | info msg if no embedded Solr http://bugs.yacy.net/view.php?id=279 Changed Files: htroot/IndexReIndexMonitor_p.java |
Sun Aug 04 05:37:20 CEST 2013 by reger | css fix for IE10 showing border on all img within <a /> tag since introduction of external link icon (commit https://gitorious.org/yacy/rc1/commit/112836dcc9bfd3966a81f2a6160569e69a256a40) Changed Files: htroot/env/base.css |
Thu Aug 01 12:36:59 CEST 2013 by Michael Peter Christen | bugfix release 1.62 Changed Files: build.properties |
Wed Jul 31 13:33:05 CEST 2013 by Michael Peter Christen | added line to enhance debugging Changed Files: htroot/Crawler_p.java |
Wed Jul 31 06:18:30 CEST 2013 by orbiter | fixed feed api servlet and and enhanced RSSReader class Changed Files: htroot/api/feed.java, source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/document/RSSReader.java |
Wed Jul 31 05:43:58 CEST 2013 by orbiter | fix for CGI test Changed Files: source/net/yacy/cora/document/MultiProtocolURI.java |
Mon Jul 29 19:09:28 CEST 2013 by sixcooler | fix for Blacklist (-Administration) Changed Files: htroot/BlacklistCleaner_p.java, htroot/BlacklistImpExp_p.java, htroot/BlacklistTest_p.java, htroot/Blacklist_p.java, htroot/IndexControlRWIs_p.java, htroot/api/blacklists.java, htroot/api/blacklists_p.java, htroot/sharedBlacklist_p.java, source/net/yacy/repository/Blacklist.java |
Sat Jul 27 15:20:09 CEST 2013 by orbiter | fix for npe Changed Files: source/net/yacy/migration.java |
Commit | Description |
---|---|
Sun Feb 09 17:37:35 CET 2014 by Michael Peter Christen | Release 1.68 Changed Files: build.properties |
Sun Feb 09 14:57:17 CET 2014 by Michael Peter Christen | extra sleep for remote searches enhances search results because there is more time for more remote peers to contribute on the first result page Changed Files: source/net/yacy/search/query/SearchEvent.java |
Sun Feb 09 14:45:12 CET 2014 by Michael Peter Christen | more efficient wordcount Changed Files: source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SearchEvent.java |
Sun Feb 09 12:36:56 CET 2014 by Michael Peter Christen | added memory protection for postprocessing Changed Files: source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Sun Feb 09 12:26:58 CET 2014 by Michael Peter Christen | removed warnings and superfluous logging Changed Files: htroot/SettingsAck_p.java, source/net/yacy/cora/document/feed/RSSFeed.java, source/net/yacy/cora/util/ConcurrentLog.java, source/net/yacy/peers/graphics/NetworkGraph.java |
Sun Feb 09 12:25:07 CET 2014 by Michael Peter Christen | removed interaction fragments Changed Files: htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/env/templates/embeddedfooter.template, htroot/env/templates/footer.template, htroot/env/templates/simplefooter.template, htroot/yacysearch.html, htroot/yacysearchitem.html, htroot/yacysearchitem.java |
Fri Feb 07 17:19:08 CET 2014 by Michael Peter Christen | fixed "Size in KBytes" calculation in PerformanceQueues_p.html, see http://bugs.yacy.net/view.php?id=362 Changed Files: htroot/PerformanceQueues_p.java, source/net/yacy/cora/util/ByteArray.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java |
Fri Feb 07 01:47:10 CET 2014 by Michael Peter Christen | removed unused classes and servlets Changed Files: |
Fri Feb 07 01:38:58 CET 2014 by Michael Peter Christen | removed libraries which had been used by jena and were added together with jena in commit 5d20cd324af387c72fedec88a8059442c090c769, see: https://gitorious.org/yacy/rc1/commit/5d20cd324af387c72fedec88a8059442c090c769 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml |
Fri Feb 07 00:22:31 CET 2014 by Michael Peter Christen | enrichment of PerformanceMemory display of SolrInfoMBean table Changed Files: htroot/PerformanceMemory_p.html, htroot/PerformanceMemory_p.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/search/index/Fulltext.java |
Fri Feb 07 00:02:19 CET 2014 by Michael Peter Christen | added statistic information for solr resources in PerformanceMemory Changed Files: htroot/PerformanceMemory_p.html, htroot/PerformanceMemory_p.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/search/index/Fulltext.java |
Thu Feb 06 23:15:00 CET 2014 by reger | fix: select all checkbox Tables_p (needs form name attribute) Changed Files: htroot/Table_YMark_p.html, htroot/Tables_p.html |
Thu Feb 06 22:39:19 CET 2014 by Michael Peter Christen | re-inserted default value re-set button to performance queues and patched missing values for recent new queues Changed Files: defaults/yacy.init, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, source/net/yacy/search/Switchboard.java |
Thu Feb 06 20:57:07 CET 2014 by Alex Vouilloz | updated and completed translation of Steering.html and ConfigUpdate_p.html into French Changed Files: locales/fr.lng |
Thu Feb 06 19:13:29 CET 2014 by Michael Peter Christen | enhanced strategy to clear solr caches - redesigned the instance mirror class (which was a mess) - added final method to close a searcher (which otherwise keeps a cache) - changed cache clear method which iterates over resources and calls clear to all caches in the searcher resources Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/search/index/Fulltext.java |
Thu Feb 06 00:46:14 CET 2014 by Alex Vouilloz | updated and completed translation of Network.html into French Changed Files: locales/fr.lng |
Wed Feb 05 19:22:59 CET 2014 by Alex Vouilloz | translated templates into French Changed Files: locales/de.lng, locales/fr.lng |
Wed Feb 05 03:45:02 CET 2014 by reger | fix not needed getFileExtension().toLower (double) add missing .getFileExtension Changed Files: htroot/yacysearchitem.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/tarParser.java |
Wed Feb 05 00:56:49 CET 2014 by Alexandre Vouilloz | completed fr.lng for Status.html Changed Files: locales/de.lng, locales/fr.lng |
Tue Feb 04 21:41:12 CET 2014 by Alexandre Vouilloz | completed fr.lng for BasicConfig.html (+modif in de.lng) Changed Files: locales/de.lng, locales/fr.lng |
Tue Feb 04 00:18:11 CET 2014 by orbiter | enhanced didyoumean Changed Files: htroot/suggest.java, source/net/yacy/cora/util/StringBuilderComparator.java, source/net/yacy/data/DidYouMean.java |
Mon Feb 03 18:49:03 CET 2014 by Michael Peter Christen | enhanced didyoumean Changed Files: source/net/yacy/data/DidYouMean.java |
Mon Feb 03 17:39:59 CET 2014 by Michael Peter Christen | reduced color depth of star icons (for smaller file sizes) Changed Files: htroot/env/grafics/star_blue.png, htroot/env/grafics/star_green.png, htroot/env/grafics/star_red.png, htroot/env/grafics/star_yellow.png |
Mon Feb 03 17:39:26 CET 2014 by Michael Peter Christen | small change to start/stop/pause icon style Changed Files: htroot/Status_p.inc, htroot/env/grafics/pause.gif, htroot/env/grafics/start.gif, htroot/env/grafics/stop.gif, htroot/js/Crawler.js |
Mon Feb 03 12:44:52 CET 2014 by orbiter | added new suggest method which replaces more-than-one suggestions: instead of computing suggest permutations of the given words, the completion of a phrase using the given words is searched in the fulltext index. Changed Files: htroot/suggest.java, source/net/yacy/data/DidYouMean.java, source/net/yacy/search/index/Segment.java |
Sun Feb 02 03:47:12 CET 2014 by reger | remove unused static var from HTTPDProxyHandler Changed Files: htroot/SettingsAck_p.java, source/net/yacy/server/http/HTTPDProxyHandler.java |
Thu Jan 30 22:57:27 CET 2014 by reger | allow combining selection of different search nav's (facets) - selecting more than one nav combines the 2 selections (with AND) - unselecting one nav clears all selected (e.g. select filetype:pdf and /language/fr shows ~ french pdf's only) Changed Files: source/net/yacy/search/query/QueryParams.java |
Thu Jan 30 22:00:32 CET 2014 by reger | fix page nav, to keeping modifier (was new issue) Changed Files: htroot/yacysearchtrailer.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java |
Thu Jan 30 10:51:08 CET 2014 by orbiter | added a boost on appearance of combined words (in the same order the user submitted that) when searching for more than one word Changed Files: source/net/yacy/search/query/QueryParams.java |
Tue Jan 28 22:39:45 CET 2014 by reger | URLproxy access check too tight respect config ip pattern (was own ip) Changed Files: source/net/yacy/http/servlets/YaCyProxyServlet.java |
Tue Jan 28 12:39:58 CET 2014 by Michael Peter Christen | - faster initialization of index files - removal of not used space if index files shrink (rare, but possible) Changed Files: source/net/yacy/kelondro/index/RowCollection.java |
Mon Jan 27 21:54:52 CET 2014 by orbiter | less logging Changed Files: htroot/NetworkPicture.java |
Mon Jan 27 13:55:06 CET 2014 by Michael Peter Christen | updated bootstrap seedlist list Changed Files: defaults/yacy.network.freeworld.unit |
Mon Jan 27 13:48:20 CET 2014 by Michael Peter Christen | enhancements for staticIP and ipv6 handling Changed Files: htroot/SettingsAck_p.java, htroot/yacy/hello.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/server/serverSwitch.java |
Mon Jan 27 01:09:56 CET 2014 by reger | add config value for ssl/https port (default=8443) adjust server routines to use config Changed Files: defaults/yacy.init, source/net/yacy/http/Jetty8HttpServerImpl.java |
Sun Jan 26 21:50:42 CET 2014 by reger | disable wrong forward to https on port change Changed Files: htroot/ConfigBasic.html |
Sun Jan 26 01:00:52 CET 2014 by reger | improve isOlder check to not overwrite node index with metadata on equal load date Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java |
Sun Jan 26 00:58:54 CET 2014 by reger | remove double jquery-1.3.1.js and adjust header links to jquery-1.3.2 Changed Files: htroot/api/ymarks/test_treeview.html, htroot/yacy/ui/index.html, htroot/yacy/ui/yacyui-widget.html, htroot/yacy/ui/yacyuisearch.html |
Sat Jan 25 22:59:23 CET 2014 by reger | add language navigation filter clause to rwi results Changed Files: source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SearchEvent.java |
Thu Jan 23 23:56:45 CET 2014 by reger | added hint to web.xml and for completeness the full set of hardcoded mappings Changed Files: defaults/web.xml, source/net/yacy/http/Jetty8HttpServerImpl.java |
Thu Jan 23 22:48:31 CET 2014 by orbiter | Added a write-enabled checkbox for remote solr servers. It is now possible to assign every peer other YaCy peers as remote solr server which are only used for read operations during search. This also affects crawling: it will exclude urls from crawls which exist on remote solr/remote YaCy peers. Changed Files: htroot/IndexFederated_p.html, htroot/IndexFederated_p.java |
Thu Jan 23 19:21:07 CET 2014 by orbiter | changed solr paths according to new default paths Changed Files: htroot/env/templates/header.template |
Thu Jan 23 19:01:31 CET 2014 by Michael Peter Christen | first try to add a generic solr servlet as luke request servlet Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/servlets/SolrSelectServlet.java, source/net/yacy/http/servlets/SolrServlet.java, source/net/yacy/search/index/Fulltext.java |
Thu Jan 23 18:41:45 CET 2014 by Michael Peter Christen | changed the web.xml as well to migrate the solr servlet Changed Files: defaults/web.xml |
Thu Jan 23 17:20:49 CET 2014 by Michael Peter Christen | refactoring: renamed the SolrServlet to SolrSelectServlet for better naming of more Solr Servlets Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/servlets/SolrSelectServlet.java |
Thu Jan 23 17:14:13 CET 2014 by Michael Peter Christen | added two virtual instances of solr for the both cores: collection1 and webgraph. These cores are now accessible at /solr/collection1/select instead /solr/select?core=collection1 and /solr/webgraph/select instead /solr/select?core=webgraph in addition to the old behavior to support compatibility to the old peers. These new paths are fully solr standard-conform and will allow the cross-linking between YaCy peers using their public solr API. Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/servlets/SolrServlet.java |
Thu Jan 23 15:56:36 CET 2014 by Michael Peter Christen | added button in host browser to re-load 404/failed documents Changed Files: htroot/HostBrowser.html, htroot/HostBrowser.java, htroot/Load_RSS_p.java, htroot/env/base.css, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/search/Switchboard.java |
Thu Jan 23 01:52:51 CET 2014 by reger | fix search navigator not showing activated nav introduced with https://gitorious.org/yacy/rc1/commit/97e84439fb749a624c00ec01a6caa076c9c1c32f Changed Files: htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java |
Wed Jan 22 21:23:32 CET 2014 by reger | optimize parse ServerSideInclude Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Wed Jan 22 17:52:07 CET 2014 by sixcooler | use less ram on dht-URL transfer by not using a URIMetadataNode[] Changed Files: source/net/yacy/peers/Protocol.java |
Wed Jan 22 17:18:53 CET 2014 by Michael Peter Christen | do not set a load prerequisite that prevents the start of one-time-jobs Changed Files: source/net/yacy/kelondro/workflow/InstantBusyThread.java |
Wed Jan 22 01:26:44 CET 2014 by sixcooler | workaround to let dht-out run on smal Systems like a Pi Changed Files: source/net/yacy/peers/Dispatcher.java |
Tue Jan 21 21:35:48 CET 2014 by reger | put list quick navigator buttons belowon BlackList_p editor replacing the dropdown -> go navigation Changed Files: htroot/Blacklist_p.html, htroot/Blacklist_p.java |
Tue Jan 21 19:28:00 CET 2014 by Michael Peter Christen | making latency_factor and maximum number of same hosts in loader queue settings available in Crawler_p.html servlet for steering. Changed Files: htroot/Crawler_p.html, htroot/Crawler_p.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/kelondro/workflow/AbstractBusyThread.java, source/net/yacy/kelondro/workflow/BusyThread.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/server/serverSwitch.java |
Tue Jan 21 17:53:39 CET 2014 by Michael Peter Christen | code simplifications / removed warnings Changed Files: source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyLoginService.java, source/net/yacy/search/Switchboard.java |
Tue Jan 21 17:52:16 CET 2014 by Michael Peter Christen | new crawling factors can now be changed during runtime Changed Files: source/net/yacy/crawler/data/Latency.java, source/net/yacy/search/Switchboard.java |
Tue Jan 21 17:48:45 CET 2014 by Michael Peter Christen | - removed hardcoded load-test which is now handled in BusyQueues steering, see /PerformanceQueues_p.html - changed default values for crawler queue load limit (high, because these jobs are started upon user request) Changed Files: defaults/yacy.init, htroot/yacy/transferRWI.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Tue Jan 21 16:56:44 CET 2014 by sixcooler | lower client-connection for single-cpu-systems Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java |
Tue Jan 21 16:05:55 CET 2014 by Michael Peter Christen | reverted crawling strategy from latest commit. Changed Files: defaults/yacy.init, htroot/PerformanceQueues_p.java, htroot/api/latency_p.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/Latency.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Mon Jan 20 21:46:40 CET 2014 by Michael Peter Christen | enhanced crawler speed Changed Files: htroot/api/latency_p.java, source/net/yacy/crawler/data/Latency.java |
Mon Jan 20 21:09:03 CET 2014 by Michael Peter Christen | added synchronization to exists() call bacause the concurrent call to that method showed in thread dump close to deadlock situations. Its also better to synchronize IO operations because they become faster then. Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Mon Jan 20 18:39:36 CET 2014 by Michael Peter Christen | if a http client thread is still running but we do not wait for it any more, call an interrupt Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java |
Mon Jan 20 18:36:00 CET 2014 by Michael Peter Christen | cleanup crawl loader jobs Changed Files: source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/search/Switchboard.java |
Mon Jan 20 08:12:35 CET 2014 by reger | fix IOexception logging issue in DefaultServlet reason not sure but .logException triggers another exception Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Mon Jan 20 00:03:55 CET 2014 by reger | upd heuristic default list (heuristicopensearch.conf) - Faroo Web taken out (requires api key) http://www.faroo.com/hp/api/api.html#description - update Faroo News to new url - Twitter taken out (change to Api 1.1 not supporting rss) https://dev.twitter.com/discussions/24239 Changed Files: defaults/heuristicopensearch.conf |
Sun Jan 19 14:47:11 CET 2014 by Michael Peter Christen | experimental removal of synchronization. This should work for all cases where the size() and isEmpty() method is used only for statistics, which happens at many locations in YaCy. If these methods are used for structual reasons (like accessing the last element in an array) then it may fail or cause other problems. As far as visible, this is not the case. Changed Files: source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/index/RAMIndex.java, source/net/yacy/kelondro/index/RAMIndexCluster.java |
Sun Jan 19 14:44:58 CET 2014 by Michael Peter Christen | removed superfluous synchronization Changed Files: source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java |
Sun Jan 19 01:50:42 CET 2014 by orbiter | added load limit for DHT transfer: RWI acceptance only if local load is not too high Changed Files: htroot/yacy/transferRWI.java, source/net/yacy/peers/Transmission.java |
Sat Jan 18 23:27:16 CET 2014 by reger | fix bug: Crawl Start (Expert) crawls "?-URLs" even if told not to do so http://bugs.yacy.net/view.php?id=329 Changed Files: htroot/Crawler_p.java |
Sat Jan 18 18:48:52 CET 2014 by reger | Skip remote Solr search if last try showed error As the solr servlet may not be available (e.g. no public search page, old version, individual access setting) a /solr/select error is remembered in the seed.dna of the remote peer. This is not permanent, as flag is not stored and the seed is reloaded on several occasions, it is just a memory of the recent past status. Might also be set to "not available" on time-out of last try. Changed Files: source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java |
Sat Jan 18 12:55:05 CET 2014 by orbiter | concurrency-solid version of transmission limitation Changed Files: source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/Protocol.java |
Sat Jan 18 12:54:14 CET 2014 by orbiter | removed -d64 jvm option because that causes problems on non-64 bit linux, see http://bugs.yacy.net/view.php?id=349 and http://bugs.yacy.net/view.php?id=339 Changed Files: addon/YaCy.app/Contents/Info.plist, addon/yacyInit.m4, startYACY.sh |
Sat Jan 18 10:50:24 CET 2014 by orbiter | reduce load during dht transfer Changed Files: source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/Protocol.java |
Fri Jan 17 14:54:44 CET 2014 by Michael Peter Christen | no commit if no action is requested Changed Files: htroot/IndexDeletion_p.java |
Fri Jan 17 14:54:19 CET 2014 by Michael Peter Christen | - better 'extra'-peer selection - logging of health status for 'extra'-peer selection - concurrency for remote peer IO and interrupting the threads if time-out occurrs Changed Files: source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java |
Fri Jan 17 09:42:55 CET 2014 by orbiter | added synchronization to deeply-embedded solr connector EmbeddedSolrConnector because deadlock situations show that methods in lucene class seem to block. Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Thu Jan 16 17:34:26 CET 2014 by Michael Peter Christen | when scaling the number of remote peers, also consider the machine load and the number of cores Changed Files: source/net/yacy/cora/util/Memory.java, source/net/yacy/peers/RemoteSearch.java |
Thu Jan 16 13:10:20 CET 2014 by Michael Peter Christen | less blocking in SeedDB which caused deadlocks in peer ping Changed Files: source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java |
Thu Jan 16 13:08:39 CET 2014 by Michael Peter Christen | better logging in logger Changed Files: source/net/yacy/cora/util/ConcurrentLog.java, source/net/yacy/kelondro/index/RowCollection.java |
Wed Jan 15 23:13:22 CET 2014 by Michael Peter Christen | replaced old caching in SolrConnector with a new one which is better for concurrency and should prevent from 100% CPU usage after a long run of a peer with a large number of documents. Changed Files: source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/cora/storage/ARH.java, source/net/yacy/cora/storage/ConcurrentARH.java |
Wed Jan 15 17:18:32 CET 2014 by Michael Peter Christen | backmigration from solrj 4.6.0 to 4.5.1. This is necessary because solrj.4.6.0 has a bug which prevents the attachment of a remote solr (as tested with a SolrCloud). See bug report https://issues.apache.org/jira/browse/SOLR-5532 This bug shall be fixed in Solr 4.6.1. Fortunately, solrj-4.5.1 works together with solr-4.6.0 thus the current index does not need to be changed. Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/solr-solrj-4.5.1.jar |
Fri Jan 10 19:31:36 CET 2014 by reger | simulate Authorization cookie for yacy servlet header Changed Files: source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Fri Jan 10 18:11:32 CET 2014 by reger | fix NPE in TemplateEngine StackTrace For input string: "" java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:504) at java.lang.Integer.parseInt(Integer.java:527) at net.yacy.server.http.TemplateEngine.writeTemplate(TemplateEngine.java:241) at net.yacy.server.http.TemplateEngine.writeTemplate(TemplateEngine.java:199) at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:896) Changed Files: source/net/yacy/server/http/TemplateEngine.java |
Fri Jan 10 14:07:49 CET 2014 by reger | implementing YaCy legacy role names - taking out customized SecurityHandler code as the original/default seems to just work fine - with this individual sec. constraints can be applied via web.xml (using legacy role names) Changed Files: source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/http/YaCyLoginService.java |
Fri Jan 10 12:38:00 CET 2014 by reger | Merge origin/master Changed Files: bin/checkindex.sh |
Fri Jan 10 12:36:42 CET 2014 by reger | make SecurityHandler webappcontext ready Changed Files: defaults/web.xml, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java |
Fri Jan 10 12:27:49 CET 2014 by Michael Peter Christen | added checkindex, solr index check Changed Files: bin/checkindex.sh |
Fri Jan 10 10:26:45 CET 2014 by reger | disabled unused fields in default Solr collection schema Changed Files: defaults/solr.collection.schema |
Fri Jan 10 10:24:33 CET 2014 by reger | spend CrawlQueues a fremem routine - clears errorStack - will not get hit often (but better little than nothing on low mem) Changed Files: source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/search/SwitchboardConstants.java |
Wed Jan 08 08:12:56 CET 2014 by reger | fix use of url proxy access pattern pattern of transparent was used. Changed Files: source/net/yacy/http/servlets/YaCyProxyServlet.java |
Tue Jan 07 22:48:45 CET 2014 by reger | Merge origin/master Changed Files: bin/apicall.sh, source/net/yacy/cora/protocol/Domains.java |
Tue Jan 07 21:58:55 CET 2014 by Michael Peter Christen | added a script which can check the solr index for inconsistencies while the peer is down. This shall be used in emergency cases where a check or fix for a broken solr index is needed. Changed Files: bin/checksolr.sh |
Tue Jan 07 21:52:19 CET 2014 by Michael Peter Christen | using the adminAccountUserName from yacy.conf within apicall.sh Changed Files: bin/apicall.sh, source/net/yacy/cora/protocol/Domains.java |
Tue Jan 07 17:53:49 CET 2014 by orbiter | introduced a new storage path ARCHIVE inside of DATA which will be used as path for solr index dumps (instead of the SEGMENTS path). This will make a maintenance of index backups easier. It will also provide a tool to migrate from an freeworld index to a webportal index. Changed Files: defaults/yacy.init, htroot/IndexControlURLs_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java |
Tue Jan 07 01:01:55 CET 2014 by sixcooler | check blacklist proxyClient config Changed Files: source/net/yacy/http/AbstractRemoteHandler.java |
Tue Jan 07 00:18:01 CET 2014 by sixcooler | make use of our DNS-cache again - this realy speeds up the lookup Changed Files: source/net/yacy/cora/protocol/Domains.java |
Mon Jan 06 18:19:54 CET 2014 by reger | better solution for prev. commit with MultiMapSolrParams.getFieldInt not returning default parameter Changed Files: source/net/yacy/http/servlets/SolrServlet.java |
Mon Jan 06 17:52:21 CET 2014 by reger | improve NPE chance on missing parameters java.lang.NullPointerException at net.yacy.http.servlets.SolrServlet.service(SolrServlet.java:145) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) Changed Files: source/net/yacy/http/servlets/SolrServlet.java |
Mon Jan 06 15:00:14 CET 2014 by reger | reimplement proxy access by configured whitlist pattern was currently limited to own ip. Changed Files: source/net/yacy/http/AbstractRemoteHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/servlets/YaCyProxyServlet.java |
Mon Jan 06 11:28:05 CET 2014 by reger | setting of IPv4Stack moved earlier it seems even better to call system.setproperty before isrunning check (if nothing helps we have to set it in startup script) Changed Files: source/net/yacy/yacy.java |
Mon Jan 06 07:22:23 CET 2014 by reger | open service manager upon install failure in installYaCyWindowsService.bat likely service is already installed Service Manager allows then to modify settings Changed Files: installYaCyWindowsService.bat |
Mon Jan 06 07:00:16 CET 2014 by reger | reimplemented server access restriction via Jetty IPAccessHandler to allow only configured IP's to access. Handler is only loaded if a restriction is configured. Since IPAcessHandler (Jetty 8) does not support IPv6 system property java.net.preferIPv4Stack=true Testing showed system.setProperty seems to be sensitive to point of calling (earliest possible time seems to be best = early in yacy.main). Moved the "isrunning..." just open browser check also to the new routine to preread the yacy.config only once. Changed Files: htroot/Settings_ServerAccess.inc, locales/de.lng, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/yacy.java |
Sun Jan 05 17:43:34 CET 2014 by Michael Peter Christen | set a realm message to log-in input window which explains that a password for the account 'admin' can be (re-)set with the script bin/passwd.sh Changed Files: source/net/yacy/data/WorkTables.java, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/YaCyLoginService.java |
Sun Jan 05 15:06:40 CET 2014 by Michael Peter Christen | cleanup Changed Files: htroot/CrawlStartScanner_p.java, source/net/yacy/http/CrashProtectionHandler.java, source/net/yacy/http/ProxyCacheHandler.java, source/net/yacy/server/http/HTTPDemon.java |
Sun Jan 05 15:04:18 CET 2014 by Michael Peter Christen | removed non-working realm setting in http client (auth for localhost was added in previous commit) Changed Files: htroot/CrawlStartScanner_p.java, htroot/Table_API_p.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/search/Switchboard.java |
Sun Jan 05 14:46:11 CET 2014 by orbiter | added http authentification to YaCy http client for all localhost acesses to enable self-steering of the peer using the API table. This is necessary in case that an password for the administration pages is set. Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/yacy.java |
Sun Jan 05 11:50:23 CET 2014 by Michael Peter Christen | modified log-in detail to enable admin-login from localhost with stored hash even if localhost access is disabled. This is urgently needed for the apicall.sh script since that is used for high-availability set-up (checkalive and indexdump for index mirroring) Changed Files: bin/apicall.sh, source/net/yacy/http/YaCyLegacyCredential.java |
Sun Jan 05 05:04:28 CET 2014 by Michael Peter Christen | made the access tracker class static because it shall be used by the jetty auth module Changed Files: htroot/AccessPicture_p.java, htroot/AccessTracker_p.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/server/serverAccessTracker.java, source/net/yacy/server/serverSwitch.java |
Sun Jan 05 04:55:30 CET 2014 by Michael Peter Christen | fixed mess with test on localhost (which means local hosts for some cases) Changed Files: source/net/yacy/cora/protocol/Domains.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/Seed.java |
Sun Jan 05 03:40:10 CET 2014 by Michael Peter Christen | removed the & explicit encoding of ampersand character since this is double-translated within the template replacement process. Changed Files: source/net/yacy/search/query/QueryParams.java |
Sat Jan 04 20:49:26 CET 2014 by reger | modified DefaultServlet parameter on invoke templates call response with post=0 (if post empty) simulating previous behavior. (template servlets typically test for post==null, found one more Crawler.p.java were empty post caused problem, = defaults not correctly set) Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Sat Jan 04 17:14:10 CET 2014 by reger | fix opensearch resultcount in yacysearch.rss see merge request https://gitorious.org/yacy/rc1/merge_requests/24 use result count in searchtrailer.xml which is on p2p search more accurate (timing) Changed Files: htroot/yacysearch.java, htroot/yacysearch.rss |
Sat Jan 04 16:10:54 CET 2014 by reger | adding proxy status to (private) status box (show also transparent and url proxy status) show search result via url proxy only if status=on Changed Files: htroot/Status.java, htroot/Status_p.inc, htroot/yacysearchitem.java |
Sat Jan 04 13:09:34 CET 2014 by reger | handle http connect for proxy refactor header cleanup (reuse existing code) Changed Files: source/net/yacy/http/ProxyCacheHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java |
Fri Jan 03 12:28:40 CET 2014 by reger | harmonize access restriction for urlproxy servlet with proxy handler, what is currently - use switched on in config - access from a local IP / hostname fix shutdown exception for crashprotection handler on interrupted connections. Changed Files: source/net/yacy/http/AbstractRemoteHandler.java, source/net/yacy/http/CrashProtectionHandler.java, source/net/yacy/http/servlets/YaCyProxyServlet.java |
Fri Jan 03 02:56:09 CET 2014 by reger | extend ssl enabled msg on status page - post the portnr Changed Files: htroot/Status.java, htroot/Status_p.inc |
Fri Jan 03 01:13:56 CET 2014 by reger | extended request for local peer check to by hostname resolved ip the current islocal() check did not detect a domain.com address as request for the local peer. Changed Files: source/net/yacy/http/AbstractRemoteHandler.java |
Thu Jan 02 09:11:08 CET 2014 by reger | add error response on deniedl proxy access send http 403 response Changed Files: source/net/yacy/http/AbstractRemoteHandler.java |
Thu Jan 02 03:37:33 CET 2014 by reger | reimplement proxy access log (to Jetty ProxyHandler) - using existing HTTPDProxyHandler logger - allow local loopback ip to access proxy Changed Files: source/net/yacy/http/AbstractRemoteHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/server/http/HTTPDProxyHandler.java |
Wed Jan 01 20:48:35 CET 2014 by reger | Security Hot-Fix: for transparent proxy. Changed Files: source/net/yacy/http/AbstractRemoteHandler.java |
Wed Jan 01 11:01:22 CET 2014 by reger | add to blacklist button in CrawlResults http://bugs.yacy.net/view.php?id=220 introduced Blacklist.add with sourcefile only parameter Changed Files: htroot/CrawlResults.html, htroot/CrawlResults.java, source/net/yacy/repository/Blacklist.java |
Wed Jan 01 10:28:49 CET 2014 by reger | moved startup execAPIActions call after Jetty startup execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, but it's more robust to place the call after http is assigned to switchboard/serverSwitch. Changed Files: source/net/yacy/search/Switchboard.java |
Wed Jan 01 10:20:32 CET 2014 by reger | change SolrServlet from Filter to Servlet (as no multicore required) this allows to simplify context/servlet initialization in Jetty init. Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/servlets/SolrServlet.java |
Tue Dec 31 23:01:41 CET 2013 by reger | fix NPE GSAresponseWriter on query=null java.lang.NullPointerException at net.yacy.cora.federate.solr.responsewriter.GSAResponseWriter.highlight(GSAResponseWriter.java:328) at net.yacy.cora.federate.solr.responsewriter.GSAResponseWriter.write(GSAResponseWriter.java:263) at net.yacy.http.servlets.SolrServlet.service(SolrServlet.java:235) Changed Files: source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java |
Tue Dec 31 15:20:34 CET 2013 by orbiter | new data type for access tracker (unfinished) Changed Files: source/net/yacy/search/query/AccessTracker.java |
Sun Dec 29 23:16:20 CET 2013 by reger | include install/uninstallYaCyWindowsService.bat as wrappers to copy Changed Files: build.xml |
Sun Dec 29 20:14:10 CET 2013 by reger | fix QueryGoal Image query (missing space) see query log example .. url_file_ext_s:(jpg OR png OR gif) ORcontent_type:(image/*)) .. Changed Files: source/net/yacy/search/query/QueryGoal.java |
Sun Dec 29 13:03:33 CET 2013 by orbiter | unescaping the path Changed Files: htroot/js/yacyinteractive.js |
Sun Dec 29 12:40:34 CET 2013 by orbiter | "inaccessible" better describes the state of a server which cannot be reached (while 30c3: too many users) Changed Files: htroot/ServerScannerList.html |
Sun Dec 29 08:05:37 CET 2013 by reger | fix NPE in IndexControlRWIs_p.html - metatags my be null Caused by: java.lang.NullPointerException at net.yacy.search.query.QueryParams.getFacets(QueryParams.java:445) at net.yacy.search.query.QueryParams.getBasicParams(QueryParams.java:400) at net.yacy.search.query.QueryParams.solrTextQuery(QueryParams.java:345) at net.yacy.search.query.QueryParams.solrQuery(QueryParams.java:334) at net.yacy.search.query.SearchEvent.<init>(SearchEvent.java:290) at net.yacy.search.query.SearchEventCache.getEvent(SearchEventCache.java:176) at IndexControlRWIs_p.genSearchresult(IndexControlRWIs_p.java:641) at IndexControlRWIs_p.respond(IndexControlRWIs_p.java:141) Changed Files: source/net/yacy/search/query/QueryParams.java |
Sun Dec 29 02:05:07 CET 2013 by reger | Merge origin/master Changed Files: htroot/IndexControlURLs_p.java, source/net/yacy/http/servlets/SolrServlet.java, source/net/yacy/search/query/AccessTracker.java |
Sun Dec 29 01:51:37 CET 2013 by reger | upd NB gui classpath Changed Files: nbproject/project.xml |
Sun Dec 29 01:39:30 CET 2013 by orbiter | switched back on index deletion (was accidently off because new jetty framework delivers never null to post arguments .. there may be more of that kind of problems) Changed Files: htroot/IndexControlURLs_p.java, source/net/yacy/http/servlets/SolrServlet.java |
Sun Dec 29 01:33:07 CET 2013 by reger | remove in distro not existing path setting from stopYaCy.bat Changed Files: stopYACY.bat |
Sun Dec 29 01:25:44 CET 2013 by orbiter | write solr searches to search log Changed Files: source/net/yacy/http/servlets/SolrServlet.java, source/net/yacy/search/query/AccessTracker.java |
Sun Dec 29 00:40:31 CET 2013 by orbiter | fix for strange NPE java.lang.NullPointerException at net.yacy.search.Switchboard.updateMySeed(Switchboard.java:3667) at net.yacy.peers.Network.peerPing(Network.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:107) at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:165) Changed Files: source/net/yacy/search/Switchboard.java |
Sat Dec 28 23:34:43 CET 2013 by orbiter | startRecord fix; this is not according to SRU definition because this states that the first record has number 0; but +1 is not consistent with other places where the number is used. Changed Files: htroot/yacysearch.java |
Sat Dec 28 22:42:02 CET 2013 by orbiter | ftp client shall be able to open non-anonymous ftp servers if login details are given Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/search/Switchboard.java |
Fri Dec 27 21:04:14 CET 2013 by reger | display configured adminUserName in ConfigAccounts_p - fix read default username in in loginservice Changed Files: htroot/ConfigAccounts_p.java, source/net/yacy/http/YaCyLoginService.java |
Fri Dec 27 06:53:14 CET 2013 by reger | Merge origin/master Changed Files: defaults/yacy.init, htroot/IndexFederated_p.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/instance/ServerShard.java, source/net/yacy/cora/federate/solr/instance/ShardInstance.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/Fulltext.java |
Fri Dec 27 06:45:22 CET 2013 by reger | tweak Jetty credentials to work with YaCy UserDB - user entry in UserDB with admin right can login to access protected pages - dto. admin user, choosen username is stored in conf (adminAccountUserName=) Changed Files: defaults/yacy.init, htroot/ConfigAccounts_p.java, htroot/SettingsAck_p.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyLoginService.java |
Fri Dec 27 03:04:14 CET 2013 by reger | Merge origin/master Changed Files: .classpath, htroot/yacysearch.java, nbproject/project.xml, source/net/yacy/document/parser/docParser.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/server/http/HTTPDFileHandler.java |
Fri Dec 27 02:59:19 CET 2013 by reger | nasty quick fix for admin login with other username as admin - userDB is not sync'ed with Jetty credentials as of now only the std. admin account can login switched initial browser open with ssl active back to std. http port Changed Files: source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyLoginService.java, source/net/yacy/yacy.java |
Fri Dec 27 02:55:21 CET 2013 by Michael Peter Christen | added option to attach remote solr servers in read-only mode Changed Files: defaults/yacy.init, htroot/IndexFederated_p.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/instance/ServerShard.java, source/net/yacy/cora/federate/solr/instance/ShardInstance.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/Fulltext.java |
Fri Dec 27 00:34:55 CET 2013 by Michael Peter Christen | add patch which shall prevent that naive search mistakes like usage of regular expressions cause no results. Usage of '*' followed by a dot or any expression will now cause that this expression is used as a filetype search. Changed Files: htroot/yacysearch.java, source/net/yacy/search/query/QueryModifier.java |
Thu Dec 26 23:59:04 CET 2013 by Michael Peter Christen | - use default files from yacy.init property "defaultFiles" if no jetty-configuration is given for default files. - fix a problem with default paths if no path is given (i.e. http://localhost:8090 instead of http://localhost:8090/). Without this patch the path was resolved automatically to http://localhost:8090// Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/server/http/HTTPDFileHandler.java |
Thu Dec 26 23:26:21 CET 2013 by Michael Peter Christen | suppress deprecation warning (for now); TODO: find alternatives Changed Files: source/net/yacy/document/parser/docParser.java |
Thu Dec 26 22:23:40 CET 2013 by sixcooler | removed classpathentry to 'remove obsolete htroot/solr htroot/gsa YaCy-servlets' Changed Files: .classpath |
Thu Dec 26 22:01:01 CET 2013 by sixcooler | missing entrys for: 'updated poi-3.9 / poi-scratchpad-3.9' Changed Files: .classpath, nbproject/project.xml |
Thu Dec 26 05:34:53 CET 2013 by reger | upd installWindowsService.bat - remove javaopts according to startYaCy.bat Changed Files: installYaCyWindowsService.bat |
Wed Dec 25 22:53:11 CET 2013 by reger | remove obsolete htroot/solr htroot/gsa YaCy-servlets - now handled by standard servlets Changed Files: |
Wed Dec 25 22:33:35 CET 2013 by reger | update Maven pom Changed Files: libbuild/pom.xml, pom.xml |
Wed Dec 25 22:31:41 CET 2013 by reger | updated poi-3.9 / poi-scratchpad-3.9 Changed Files: addon/YaCy.app/Contents/Info.plist, build.xml, lib/poi-3.9-20121203.License, lib/poi-3.9-20121203.jar, lib/poi-scratchpad-3.9-20121203.License, lib/poi-scratchpad-3.9-20121203.jar |
Wed Dec 25 05:33:00 CET 2013 by reger | run SSL/HTTPS port (8443) ping test in migration only if SSL/HTTPS is on - see last commit Changed Files: source/net/yacy/migration.java |
Tue Dec 24 05:11:31 CET 2013 by reger | remove obsolete HTTPServer input field Changed Files: htroot/Settings_Http.inc, htroot/Settings_p.java |
Mon Dec 23 12:05:44 CET 2013 by Michael Peter Christen | aaaaaaaaaaaaaa Changed Files: addon/yacyInit.m4 |
Mon Dec 23 11:20:52 CET 2013 by Michael Peter Christen | remove large memory pages option and added MMapDirectoryFactory for debian Changed Files: addon/yacyInit.m4 |
Mon Dec 23 02:06:33 CET 2013 by Michael Peter Christen | added html retrieval from alternative DATA/HTDOCS path Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/server/http/HTTPDFileHandler.java |
Mon Dec 23 01:51:48 CET 2013 by Michael Peter Christen | removed more unused code Changed Files: source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java |
Mon Dec 23 01:32:29 CET 2013 by Michael Peter Christen | refactoring: moving all servlets to the same package (the solr servlet is currently actually a filter which should be changed somehow) Changed Files: htroot/solr/select.java, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/servlets/GSAsearchServlet.java, source/net/yacy/http/servlets/SolrServlet.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/http/servlets/YaCyProxyServlet.java |
Fri Dec 20 00:45:58 CET 2013 by Michael Peter Christen | solr servlet preparation for join with jetty branch Changed Files: htroot/solr/select.java, source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/server/serverObjects.java |
Thu Dec 19 22:21:33 CET 2013 by Michael Peter Christen | updated guava to 15.0 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/guava-15.0.jar |
Thu Dec 19 22:16:01 CET 2013 by Michael Peter Christen | upgraded json-simple to 1.1.1 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/json-simple-1.1.1.License, lib/json-simple-1.1.1.jar |
Thu Dec 19 22:13:34 CET 2013 by Michael Peter Christen | upgraded jsch to 0.1.50 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/jsch-0.1.50.License, lib/jsch-0.1.50.jar |
Thu Dec 19 22:09:01 CET 2013 by Michael Peter Christen | removed mysql connector which was used only for testing in the past Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml |
Thu Dec 19 22:06:10 CET 2013 by Michael Peter Christen | migrated to pdfbox 1.8.3 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/fontbox-1.8.3.License, lib/fontbox-1.8.3.jar, lib/jempbox-1.8.3.License, lib/jempbox-1.8.3.jar, lib/pdfbox-1.8.3.License, lib/pdfbox-1.8.3.jar |
Thu Dec 19 10:13:43 CET 2013 by Michael Peter Christen | added patch to allow binary response writer Changed Files: source/net/yacy/cora/federate/solr/SolrServlet.java |
Thu Dec 19 09:42:01 CET 2013 by Michael Peter Christen | update to Mac start parameters Changed Files: addon/YaCy.app/Contents/Info.plist |
Thu Dec 19 09:07:04 CET 2013 by Michael Peter Christen | removed malicious java start arguments in debian start script which had been removed from the other start scripts recently. The reason to do so was a comparisment of a debian-installed YaCy with 20 million document which crashed after 10 hours with the debian start script, but did not crash with the startYACY.sh start script. Both scripts now use the same java start arguments. Added also the Solr MMapDirectoryFactory switch which was missing so far in the debian start script. Changed Files: addon/yacyInit.m4 |
Thu Dec 19 09:04:23 CET 2013 by Michael Peter Christen | removed non-root restriction in startYACY.sh because that is also not applied for the debian version. There is no reason to apply this restriction. Added also the -d64 option if the system is 64 bit. Changed Files: startYACY.sh |
Tue Dec 17 14:02:27 CET 2013 by orbiter | added parsing of metadata to surrogate reading: a dublin core record inside of surrogate input files may now contain tokens within the namespace 'md' (short for: metadata). The token names must be valid withing the namespace of the solr field names. All md-tokens inside of surrogate files then overwrite values within solr documents before they are written to the solr index. This makes it possible to assign collection names to each surrogate entry and also ranking information can be added. Please see the example file. Changed Files: examples/surrogate_dublin_core.xml, source/net/yacy/document/content/SurrogateReader.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Sun Dec 15 23:39:19 CET 2013 by reger | remove unused init parameter from DefaultServlet - remove "RelativeResourceBase" parameter Changed Files: source/net/yacy/http/YaCyDefaultServlet.java |
Sun Dec 15 19:12:49 CET 2013 by orbiter | using stringbuilder in query construction Changed Files: source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java |
Sat Dec 14 19:50:41 CET 2013 by reger | update Netbeans GUI build classpath Changed Files: nbproject/project.xml |
Thu Dec 12 11:50:48 CET 2013 by Michael Peter Christen | use pre-compiled patterns in ymark Changed Files: htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/get_xbel.java, source/net/yacy/data/ymark/YMarkDMOZImporter.java, source/net/yacy/data/ymark/YMarkTables.java, source/net/yacy/data/ymark/YMarkUtil.java |
Thu Dec 12 10:55:36 CET 2013 by Michael Peter Christen | made pattern computation static Changed Files: source/net/yacy/cora/document/id/DigestURL.java |
Thu Dec 12 10:55:12 CET 2013 by Michael Peter Christen | added date parser caches to prevent re-calculation of costly date parsing Changed Files: source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/ResponseHeader.java |
Thu Dec 12 03:37:19 CET 2013 by Michael Peter Christen | counting search requests in solr interface Changed Files: htroot/solr/select.java, htroot/yacy/seedlist.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/search/query/SearchEvent.java |
Thu Dec 12 03:36:30 CET 2013 by Michael Peter Christen | avoid usage of existsByQuery. If a document can be loaded by the ID before testing other fields from the existsByQuery request, then a document cache fills and queries after that one can be avoided. Changed Files: source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/CachedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/index/ErrorCache.java |
Mon Dec 09 23:45:57 CET 2013 by reger | join DefaultServlet and Jetty8 implementation - removing Jetty 8 specific dependencies Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/YaCyDefaultServlet.java |
Mon Dec 09 23:43:58 CET 2013 by reger | update Maven pom to latest version number - include newer dependency versions of several lib/jar for eval. Changed Files: pom.xml |
Sun Dec 08 00:56:45 CET 2013 by reger | move conditionalHeader to DefaultServlet - by removing Jetty specific implementation detail Changed Files: source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Fri Dec 06 14:30:47 CET 2013 by Michael Peter Christen | added more properties to seedlist servlet Changed Files: htroot/yacy/seedlist.java |
Fri Dec 06 13:47:50 CET 2013 by Michael Peter Christen | next development version Changed Files: build.properties |
Thu Dec 05 00:55:35 CET 2013 by Michael Peter Christen | changed start parameters which caused deadlocks in mac and windows versions Changed Files: addon/YaCy.app/Contents/Info.plist, startYACY.bat |
Wed Dec 04 23:16:06 CET 2013 by Michael Peter Christen | intermediate release 1.66 Changed Files: build.properties |
Wed Dec 04 23:15:10 CET 2013 by Michael Peter Christen | better calibration of caches and queue maximum sizes Changed Files: source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SearchEvent.java |
Wed Dec 04 22:44:51 CET 2013 by Michael Peter Christen | perform greedy learning heuristic only if load < 1.0 Changed Files: htroot/yacysearchitem.java |
Wed Dec 04 18:13:43 CET 2013 by Michael Peter Christen | changed default timeout for port scanner Changed Files: htroot/CrawlStartScanner_p.html |
Wed Dec 04 17:58:36 CET 2013 by Michael Peter Christen | removed unnecessary check which causes a NPE when searching with empty search string Changed Files: source/net/yacy/search/query/QueryParams.java |
Wed Dec 04 17:48:12 CET 2013 by Michael Peter Christen | - reduce computation in case that specific postprocessing fields are not selected - de-select citation rank computation Changed Files: defaults/solr.collection.schema, defaults/solr.webgraph.schema, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Wed Dec 04 16:04:29 CET 2013 by Michael Peter Christen | removed optimization bevore postprocessing because that may cause a time-out which will cause that postprocessing fails. Changed Files: source/net/yacy/search/Switchboard.java |
Wed Dec 04 13:09:15 CET 2013 by Michael Peter Christen | removed filter computation in Protocol class for remote searches because that is already done in the QueryParams class Changed Files: source/net/yacy/peers/Protocol.java |
Wed Dec 04 12:59:30 CET 2013 by Michael Peter Christen | - performance graph does not shop peer ping in memory monitor any more - after a forced GC, the PerformanceMemory view switches to automatic update by default Changed Files: htroot/PerformanceGraph.java, htroot/PerformanceMemory_p.html, htroot/PerformanceMemory_p.java, htroot/Performance_p.html, source/net/yacy/peers/graphics/ProfilingGraph.java |
Sun Dec 01 19:36:24 CET 2013 by reger | clean-up Jetty handler classes Changed Files: source/net/yacy/http/ProxyHandler.java |
Sun Dec 01 03:38:58 CET 2013 by reger | clean-up test cases to work with current source Changed Files: test/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnectorTest.java |
Sat Nov 30 06:29:57 CET 2013 by reger | add read queryparameter in gsaservlet Changed Files: source/net/yacy/http/GSAsearchServlet.java, source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Fri Nov 29 22:10:14 CET 2013 by reger | remove reference to solrServlet from YaCy servlet select - reference is not used - solrServlet is used in Jetty branch and adjustments there conflict with unused solrServlet here. Changed Files: htroot/solr/select.java |
Fri Nov 29 02:40:12 CET 2013 by sixcooler | tried some hardening in order not letting any Solr-Searchers open Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Fri Nov 29 01:28:48 CET 2013 by Michael Peter Christen | setting of some default values to make an client development start easy using the description at http://www.yacy-websuche.de/wiki/index.php/Dev:APIhello Changed Files: htroot/yacy/hello.java |
Thu Nov 28 01:31:53 CET 2013 by Michael Peter Christen | tried to fix timeout and connection-lost problems when using an outside solr. Changed Files: defaults/yacy.init, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/search/Switchboard.java |
Thu Nov 28 01:10:25 CET 2013 by Michael Peter Christen | try to recover from an OOM during citation index reading and fail-over to second solr core in case of unrecoverable OOM. Changed Files: source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/search/index/Segment.java |
Thu Nov 28 00:54:53 CET 2013 by Michael Peter Christen | fixed a problem with Date fields parsing Solr results if a remote Solr is attached. Changed Files: source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/search/index/Fulltext.java |
Wed Nov 27 19:07:36 CET 2013 by sixcooler | memory-leak-fix: the DocListSearcher fires an query in its constructor and it is highly recommend to close every SolrRequest. Every Request, which is not closed leaves a Searcher with its Chaches an can not be garbage-collectet. Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Wed Nov 27 02:51:02 CET 2013 by reger | implement core selection in SolrServlet - making initcore() obsolete Changed Files: source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/http/Jetty8HttpServerImpl.java |
Tue Nov 26 07:06:25 CET 2013 by Michael Peter Christen | disallow remote robots to crawl the HostBrowser servlet Changed Files: htroot/robots.txt |
Tue Nov 26 07:03:20 CET 2013 by Michael Peter Christen | only admins should be able to force a commit Changed Files: htroot/HostBrowser.java |
Tue Nov 26 02:24:47 CET 2013 by Michael Peter Christen | fixes for searches containing stopwords. The fix was done using a reconstruction of the search word set access method to protect that words are deleted from the sets from the outside of the QueryGoal class. Changed Files: htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Mon Nov 25 23:30:35 CET 2013 by Michael Peter Christen | hack to remove compiler warnings about deprecated classes. It would be better to remove the deprecated usage but to do this the Solr core must adopt the latest apache http core changes as well .. this is not our fault. Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Mon Nov 25 21:31:40 CET 2013 by orbiter | using the BinaryResponseWriter which is supported within the YaCy solr servlet since YaCy 1.63. This is much more performant for the client than using the XMLResponseWriter because parsing of XML data is very CPU intensive. Older YaCy peers are still requested using the XMLResponseWriter but the majority of YaCy peers already respond with the binary writer. This makes remote searches much faster and less CPU intensive. Changed Files: source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/index/Fulltext.java |
Mon Nov 25 15:20:54 CET 2013 by orbiter | less word hash computations (removing some overhead because of MD5 calcs) using the clear word in a normalized form. Changed Files: htroot/yacysearch.java, source/net/yacy/document/Condenser.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Mon Nov 25 00:14:53 CET 2013 by reger | add check to prevent index entries containing url_file_ext_s with ";jsession=xyz" note: check could be implemented in MultiProtocolURL (but at this time didn't oversee possible implication) Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Sun Nov 24 20:11:39 CET 2013 by reger | remove unused options and attributes from DefaultServlet cleanup obsolete class files Changed Files: source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Sun Nov 24 04:31:08 CET 2013 by reger | fix contentType definition for Solr html responswriter from xml to html (hint: value is currently not used, but is in SolrServlet) Changed Files: source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Fri Nov 22 14:15:31 CET 2013 by Michael Peter Christen | refactoring of seed attributes (introduced more constants) Changed Files: htroot/Blog.java, htroot/BlogComments.java, htroot/Wiki.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Seed.java |
Fri Nov 22 10:38:42 CET 2013 by Michael Peter Christen | added test to Base64Order (runs successfully!) Changed Files: source/net/yacy/cora/order/Base64Order.java |
Fri Nov 22 09:53:32 CET 2013 by Michael Peter Christen | fir for XSS Changed Files: htroot/HostBrowser.java |
Tue Nov 19 15:56:10 CET 2013 by orbiter | added new servlet which generates the same file as the principal peers upload to a bootstrap position you can call it either with http://localhost:8090/yacy/seedlist.html or to generate json (or jsonp) with http://localhost:8090/yacy/seedlist.json http://localhost:8090/yacy/seedlist.json?callback=seedlist Changed Files: htroot/yacy/seedlist.html, htroot/yacy/seedlist.java, htroot/yacy/seedlist.json, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java |
Mon Nov 18 19:41:31 CET 2013 by reger | for searchresults with empty title use filename as title - to not store a title in index which isn't extracted from source the title is empty check only added to ResultEntry class Changed Files: source/net/yacy/search/snippet/ResultEntry.java |
Mon Nov 18 03:15:45 CET 2013 by reger | - disable Jetty servlet defaultUseCache (prevent double caching) - include short memory status check for class cache in DefaultServlet - remove obsolete Resource interface for Jetty8YaCyDefaultServlet Changed Files: source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Sun Nov 17 00:02:08 CET 2013 by reger | use writeOptionHeaders with ServletResponse parameter only Changed Files: source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Sat Nov 16 18:30:06 CET 2013 by orbiter | extended also timeout fr webgraph postprocessing Changed Files: source/net/yacy/search/schema/WebgraphConfiguration.java |
Sat Nov 16 18:29:08 CET 2013 by orbiter | extended timeout during postprocessing of 30 minutes. Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Sat Nov 16 18:23:14 CET 2013 by orbiter | more monitoring for postprocessing and enhanced layout in Crawler monitor page Changed Files: htroot/Crawler_p.html, htroot/Crawler_p.java, htroot/api/status_p.java, htroot/api/status_p.xml, htroot/js/Crawler.js, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/search/Switchboard.java |
Sat Nov 16 08:23:42 CET 2013 by Michael Peter Christen | more monitoring for postprocessing Changed Files: htroot/Crawler_p.html, htroot/api/status_p.java, htroot/api/status_p.xml, htroot/js/Crawler.js, source/net/yacy/search/index/Segment.java |
Fri Nov 15 15:41:41 CET 2013 by Michael Peter Christen | prevent requesting of 0-facets (which actually exist) Changed Files: source/net/yacy/search/query/QueryParams.java |
Fri Nov 15 15:40:07 CET 2013 by Michael Peter Christen | enhanced mass storage speed to remote solr servers Changed Files: source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java |
Fri Nov 15 15:38:00 CET 2013 by Michael Peter Christen | added an optimization option to HandleSet mass data storage structure Changed Files: source/net/yacy/cora/storage/HandleSet.java, source/net/yacy/kelondro/index/RowHandleSet.java |
Thu Nov 14 01:37:51 CET 2013 by reger | - DefaultServlet: remove static gzip option YaCy doesn't use pre-gzip'ed static html pages - ProxyServlet: remove not neede procedure - Server init: skip one overlaping servlet context Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java, source/net/yacy/http/YaCyProxyServlet.java |
Wed Nov 13 06:04:28 CET 2013 by Michael Peter Christen | logger should be static Changed Files: source/net/yacy/visualization/RasterPlotter.java |
Tue Nov 12 15:54:54 CET 2013 by Michael Peter Christen | fix for search requests in GSA interface which contain 'funny' characters (like ':' etc.) Changed Files: htroot/gsa/searchresult.java, source/net/yacy/search/query/QueryGoal.java |
Mon Nov 11 11:25:01 CET 2013 by orbiter | replaced old existsByIds for embedded Solr with obviously much faster new selection method (including stil existing debug code to test that this is in fact better) Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Sun Nov 10 22:20:35 CET 2013 by reger | make test directory available in Maven pom - exclude reference to old slf4j-log4j12 Changed Files: pom.xml |
Sun Nov 10 22:01:31 CET 2013 by reger | cleanup test directory from Jetty 9 implementation samples - current Jetty implementation advances so that it seems not beneficial to keep the code as it makes the test unuseable and use of Jetty 9 is due to Java 1.7 dependency not in sight. Changed Files: test/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnectorTest.java |
Sun Nov 10 18:59:18 CET 2013 by reger | implement Jetty8HttpServerImpl.generateSocketAddress (code 1:1 copied from serverCore) Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/server/serverCore.java |
Sun Nov 10 18:50:34 CET 2013 by orbiter | added (partly commented) test code for url rewrite methods .. to be completed Changed Files: source/net/yacy/document/Document.java, source/net/yacy/search/Switchboard.java |
Sun Nov 10 18:48:00 CET 2013 by orbiter | better default value for crawler user agent Changed Files: htroot/Crawler_p.java |
Sun Nov 10 01:36:57 CET 2013 by reger | add highlight queryparams to solrservlet if missing - modify query params in Solr parameter map (instead of querystring) Changed Files: source/net/yacy/cora/federate/solr/SolrServlet.java |
Sat Nov 09 01:43:44 CET 2013 by Michael Peter Christen | reverted autowarming setting in solrconfig Changed Files: defaults/solr/solrconfig.xml |
Thu Nov 07 21:44:14 CET 2013 by reger | Merge origin/master into jetty Conflicts: source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java Changed Files: source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Thu Nov 07 12:13:12 CET 2013 by Michael Peter Christen | calculating and showing the number of files and the average size of a file in the HTCACHE in ConfigHTCache_p.html Changed Files: htroot/ConfigHTCache_p.html, htroot/ConfigHTCache_p.java, source/net/yacy/crawler/data/Cache.java |
Thu Nov 07 11:57:01 CET 2013 by Michael Peter Christen | fixed another memory leak in the PDF parser: the class org.apache.pdfbox.pdmodel.font.PDFont occupies 8MB of space which cannot be cleaned if PDFont.clearResources is called. The attempt to clean the class cache therefore causes that the class is loaded and this cache is initialized with some rubbish. I tried to prevent to instantiate this class by usage of a hacked findLoadedClass call to the SystemClassLoader (which is protected ...). Now, without using the PDF parser at all, 8MB of RAM space is not occupied, however, when the first PDF arrives this space will be taked and never given back to GC. WAKE UP YOU LAZY PDFBOX HACKER AND FIX THIS SHIT! Changed Files: source/net/yacy/document/parser/pdfParser.java |
Thu Nov 07 10:55:02 CET 2013 by Michael Peter Christen | reduced footprint of BookmarksDB which can take quite a lot of memory if the number of bookmarks is high (i.e. > 2000 URLs) Changed Files: source/net/yacy/data/BookmarksDB.java |
Thu Nov 07 03:11:03 CET 2013 by reger | add content_type:image/* to image search - see numerous idx entries with content_type image without url_file_ext_s (for various reason) which should be included in result - try it yourself with following sample query /solr/select?q=content_type:image/* AND -url_file_ext_s:[* TO *]&defType=edismax&fl=sku,url_file_ext_s,content_type adresses also possible url without or deviating extension. Changed Files: source/net/yacy/search/query/QueryGoal.java |
Thu Nov 07 00:32:21 CET 2013 by reger | move writeHeaders from Jetty8 servlet to YaCyDefaultServlet - after removing Jetty server dependency (of Response using HttpServletResponse only) Changed Files: source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Wed Nov 06 18:05:46 CET 2013 by Michael Peter Christen | replaced referrer link masking to 'pure' links to the referring page (that was more useful during testing) Changed Files: htroot/HostBrowser.java |
Wed Nov 06 18:05:02 CET 2013 by Michael Peter Christen | added missing unicode transformation in href link contents during parsing Changed Files: source/net/yacy/document/parser/html/ContentScraper.java |
Wed Nov 06 15:01:40 CET 2013 by Michael Peter Christen | added clickdepth and CR computation debug code to verify that the process is complete Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Wed Nov 06 15:00:57 CET 2013 by Michael Peter Christen | we do not need the load-image flag any more since this is now controlled by parser switches Changed Files: defaults/yacy.init |
Tue Nov 05 22:57:55 CET 2013 by reger | add AccessTracker logging to SolrServlet Changed Files: source/net/yacy/cora/federate/solr/SolrServlet.java |
Mon Nov 04 21:11:12 CET 2013 by reger | implement HtmlResponseWriter to solrServlet (and rss / opensearch responswriter) as in yacy select servlet. - set contenttype of HTLM/GrepHTML-Reponsewriter to "text/html" - set a contenttype to GSAsearchServlet Changed Files: source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/http/GSAsearchServlet.java |
Mon Nov 04 11:59:28 CET 2013 by Michael Peter Christen | load image only if their parser flag is activated Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Mon Nov 04 09:37:31 CET 2013 by Michael Peter Christen | less overhead calling exist() with only one hash Changed Files: source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/peers/Transmission.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java |
Mon Nov 04 03:12:02 CET 2013 by reger | fix html header in Solr HTMLResponseWriter - move 1st body content after </head> tag - add closing <span> tag Changed Files: source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Sun Nov 03 21:34:21 CET 2013 by reger | implement a Jetty reconnect to work with Configbasic_p.html port change - instead of shutting down the server it should be sufficient to manipulate the Jetty http connector Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java |
Sun Nov 03 18:31:50 CET 2013 by Michael Peter Christen | avoid cloning Changed Files: source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/ConcurrentUpdateSolrConnector.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/search/index/Fulltext.java |
Sun Nov 03 02:38:36 CET 2013 by reger | upd Maven pom (take back introduced java-templates) to handle filtering of yacyBuildProperties.java. To keep it compatible with ant filter directly from original sourcd/.... location. Changed Files: pom.xml |
Sun Nov 03 02:26:00 CET 2013 by reger | fix servlet class exist check to use default path only (in Jetty8YaCyDefaultServlet) - del redundant doget code in yacydefaultservlet - small declaration code opts - del obsolete libt/proxyservlet.java Changed Files: source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Sat Nov 02 06:27:18 CET 2013 by reger | update Maven build script - use current YaCy version number - make use of libbuild\GitRevMavenTask (maven-plugin-gitrevisionnumber) - make yacyBuildProperties.java available for source filtering by Maven-plugin (copy to libbuild\java-templates) - update assembly definition to include lib\yacycore.jar without version number (needed this way by startupscript) Changed Files: assembly.xml, libbuild/java-templates/net/yacy/peers/operation/yacyBuildProperties.java, pom.xml |
Sat Nov 02 02:33:06 CET 2013 by reger | add Maven plugin to return a YaCy style Git repository build release number and timestamp - it injects properties which can be used in pom via ${DSTAMP} ${releaseNr} if added as plugin via <plugin> <groupId>net.yacy</groupId> <artifactId>maven-plugin-gitrevisionnumber</artifactId> <version>1.0</version> <executions><execution> <phase>initialize</phase> <goals><goal>create</goal></goals> </execution></executions> </plugin> Changed Files: libbuild/GitRevMavenTask/GitRevMavenTask.java, libbuild/pom.xml |
Fri Nov 01 13:30:53 CET 2013 by Michael Peter Christen | in host browser also show the properties of failed documents including referrer urls (this is a VERY USEFUL SEO and Web Admin feature!!) Changed Files: htroot/HostBrowser.java, source/net/yacy/search/index/Segment.java |
Thu Oct 31 16:20:33 CET 2013 by orbiter | version number change for next development cycle Changed Files: build.properties |
Thu Oct 31 16:16:29 CET 2013 by orbiter | use html encoding for urls in metadata Changed Files: htroot/HostBrowser.java |
Wed Oct 30 23:11:36 CET 2013 by reger | - add GSA search /gsa/search servlet for Jetty to Server init - include SecurityHandler check for /gsa/ /solr/ - change one more YaCyDefaultServlet dependency from Jetty to std. javax.Servlet Changed Files: source/net/yacy/http/GSAsearchServlet.java, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/Jetty8YaCySecurityHandler.java |
Wed Oct 30 01:13:12 CET 2013 by reger | update Maven pom and jar dependencies Changed Files: pom.xml |
Wed Oct 30 00:51:04 CET 2013 by reger | reduce logging (by assigning logger to existing logger) - small additional cleanups Changed Files: source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/TemplateHandler.java, source/net/yacy/http/YaCyDefaultServlet.java, source/net/yacy/http/YaCyProxyServlet.java |
Sun Oct 27 19:15:20 CET 2013 by reger | reactivate Domain handler for .yacy / .yacyh handling Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/YacyDomainHandler.java |
Sun Oct 27 15:24:04 CET 2013 by reger | resolve merge conflict - add missing import statement Changed Files: source/net/yacy/yacy.java |
Sun Oct 27 15:12:17 CET 2013 by reger | Merge origin/master into jetty - merge conflict resolve Changed Files: source/net/yacy/yacy.java |
Sun Oct 27 05:04:24 CET 2013 by reger | add YaCyProxyServlet to handle /proxy.html?url=proxyurl - based on Jetty ProxyServlet - at this time use existing HTTPD ProxyHandler for url rewrite - add jetty-client jar (dependency in Jetty ProxyServlet) reuse ProxyHandler.convertHeaderFromJetty in YaCyDefaultServlet Changed Files: build.xml, lib/jetty-client-8.1.13.v20130916.jar, libt/ProxyServlet.java, source/net/yacy/http/Jetty8HttpServerImpl.java, source/net/yacy/http/YaCyDefaultServlet.java, source/net/yacy/http/YaCyProxyServlet.java |
Sun Oct 27 04:42:33 CET 2013 by reger | catch one more malformed url in proxy url rewrite Changed Files: source/net/yacy/server/http/HTTPDFileHandler.java |
Sun Oct 27 04:06:11 CET 2013 by reger | catch proxy url rewrite exception malformed url (" http:\/\/" ) may cause error response testcase http://localhost:8090/proxy.html?url=http://dictionary.reference.com/browse/test Changed Files: source/net/yacy/server/http/HTTPDFileHandler.java |
Sat Oct 26 01:31:26 CEST 2013 by orbiter | next intermediate release 1.64 Changed Files: build.properties |
Sat Oct 26 00:34:55 CEST 2013 by Michael Peter Christen | - fixed opensearchdescription, this delivered an url with missing 'global' option - added display=2 to compare_yacy to remove the superfluous border Changed Files: htroot/opensearchdescription.java, htroot/opensearchdescription.xml |
Sat Oct 26 00:33:36 CEST 2013 by Michael Peter Christen | integrated startpage to compare_yacy Changed Files: defaults/yacy.init, htroot/compare_yacy.html, htroot/compare_yacy.java |
Sat Oct 26 00:15:25 CEST 2013 by Michael Peter Christen | added boolean load property to CacheResource_p servlet which causes that the servlet loads the page from the web. Changed Files: htroot/CacheResource_p.java |
Thu Oct 24 20:42:34 CEST 2013 by orbiter | - removed a lot of garbage and bloated code from GuiHandler. - transformed log lines to String before they are stored because the storage space is about 1:250 (45kb for one line before transformation, 180 bytes afterwards) - this saves up to 10MB RAM so we can increase the number of lines to 1000 again. Changed Files: defaults/yacy.logging, source/net/yacy/kelondro/logging/GuiHandler.java |
Thu Oct 24 17:39:50 CEST 2013 by Michael Peter Christen | Automatically flush the log cache if a short memory status is reached. For the default of 200 lines this can flush about 10MB. Changed Files: htroot/ViewLog_p.java, source/net/yacy/kelondro/logging/GuiHandler.java, source/net/yacy/search/Switchboard.java |
Thu Oct 24 16:20:20 CEST 2013 by Michael Peter Christen | Enhanced crawl start for very, very large crawl lists (i.e. > 5000) which had a problem because of badly used concurrency. This fix also caused a redesign of the whole host deletion process. This should fix bug http://bugs.yacy.net/view.php?id=250 Changed Files: htroot/CrawlResults.java, htroot/Crawler_p.java, htroot/IndexControlURLs_p.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/index/Fulltext.java |
Thu Oct 24 16:17:53 CEST 2013 by Michael Peter Christen | reduced logging line memory, 10000 lines had filled up 450MB! grrr. (thank you, a bomb from the past) Changed Files: defaults/yacy.logging, source/net/yacy/kelondro/logging/GuiHandler.java |
Thu Oct 24 15:48:05 CEST 2013 by Michael Peter Christen | enhanced computation of number of solr documents. Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java |
Wed Oct 23 11:27:19 CEST 2013 by Michael Peter Christen | added concurrency for mass crawl check Changed Files: htroot/CrawlCheck_p.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/crawler/robots/RobotsTxt.java |
Tue Oct 22 15:34:51 CEST 2013 by Michael Peter Christen | When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting. Changed Files: source/net/yacy/search/Switchboard.java |
Tue Oct 22 14:20:17 CEST 2013 by Michael Peter Christen | replaced old tag cloud and adopted design a bit Changed Files: htroot/ConfigSearchPage_p.html, htroot/env/base.css, htroot/yacy/ui/js/jquery.tagcloud.js, htroot/yacysearch.html, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java |
Tue Oct 22 12:46:10 CEST 2013 by Michael Peter Christen | added matching in url field in Table_API_p search Changed Files: htroot/Table_API_p.html, htroot/Table_API_p.java |
Mon Oct 21 15:16:29 CEST 2013 by Michael Peter Christen | added hint to development/commit rss feed Changed Files: htroot/ConfigUpdate_p.html, source/net/yacy/search/Switchboard.java |
Mon Oct 21 15:10:58 CEST 2013 by Michael Peter Christen | Show a RSS icon in the right top corner of search results. This replaces the 'API' icon which was the link for the opensearch result which is an extension of RSS. Since it is more appropriate to visualize a RSS link with an RSS icon, this API icon was changed here. Changed Files: htroot/env/grafics/rss.png, htroot/yacysearch.html |
Fri Oct 18 14:14:13 CEST 2013 by orbiter | - prevent that crawl filter have empty (never-match) content - rewrite the description of the options "Restrict to start domain(s)" and "Restrict to sub-path(s)" to an explanation, that the restriction applies to all links in the link list of the option "From Link-List of URL" if this option is selected - allow "Restrict to sub-path(s)" if the "From Link-List of URL" is selected. This is supported in the crawl start. Changed Files: htroot/CrawlStartExpert_p.html |
Fri Oct 18 13:55:37 CEST 2013 by orbiter | fix for mustmatch regex computation: result had correct semantic, but may have contained multiple same expressions within the disjunction of domain-restrictions. This fix removes the redundant restrictions and makes the regex shorter. Changed Files: source/net/yacy/crawler/data/CrawlProfile.java |
Fri Oct 18 00:42:38 CEST 2013 by reger | add graceful Jetty shutdown option - as Jetty stop is not synced, yet - include jetty jars and servlet-3.0 api jar in Eclipse .classpath Changed Files: .classpath, source/net/yacy/http/Jetty8HttpServerImpl.java |
Thu Oct 17 18:36:35 CEST 2013 by orbiter | make it easier to filter properties (case insensitive) Changed Files: htroot/ConfigProperties_p.html |
Thu Oct 17 03:34:22 CEST 2013 by reger | allow to choose used http server, YaCy-Anomic or Jetty - defaults to Jetty (in this branch) - add server version info & config option -> Admin Console -> Advanced Settings -> Http Networking Changed Files: defaults/yacy.init, htroot/Settings_Http.inc, htroot/Settings_p.java, source/net/yacy/yacy.java |
Thu Oct 17 00:06:17 CEST 2013 by reger | add YaCy HttpCommand "authenticate" check to DefaultServlet Changed Files: source/net/yacy/http/YaCyDefaultServlet.java |
Wed Oct 16 11:27:06 CEST 2013 by Michael Peter Christen | enhanced postprocessing: fixed bugs, enable proper postprocessing also without the harvestingkey, remove crawl profiles after postprocessing, speed-up for clickdepth computation. Changed Files: source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Wed Oct 16 02:25:39 CEST 2013 by reger | make use of declared static string ACTION_LOCATION Changed Files: htroot/Blacklist_p.java, htroot/BlogComments.java, htroot/Connections_p.java, htroot/Status.java, htroot/User.java, htroot/Wiki.java, htroot/sharedBlacklist_p.java |
Wed Oct 16 01:48:44 CEST 2013 by reger | add YaCy HttpCommand "location" check to DefaultServlet Changed Files: source/net/yacy/http/YaCyDefaultServlet.java |
Tue Oct 15 16:46:59 CEST 2013 by Michael Peter Christen | more methods for the table class Changed Files: source/net/yacy/kelondro/blob/Tables.java |
Tue Oct 15 16:46:06 CEST 2013 by Michael Peter Christen | Replaced the inframe loading from yacy.net for donations with the loading of this iframe from the local host. To make this more flexible, this iframe is loaded once after startup from yacy.net. Changed Files: defaults/yacy.init, source/net/yacy/yacy.java |
Tue Oct 15 03:25:24 CEST 2013 by reger | remove wrong content mod in SSI parser for virtual path /currentyacypeer/ (is handled on start of request handling) Changed Files: source/net/yacy/http/YaCyDefaultServlet.java |
Mon Oct 14 04:18:52 CEST 2013 by reger | add support for gzip encoded multipart forms (needed for transferRWI.html) - quick and dirty reuse of existing HTTPDemon implementation Changed Files: source/net/yacy/http/YaCyDefaultServlet.java |
Sun Oct 13 20:56:03 CEST 2013 by reger | - add size check to multipart form data handling of YaCyDefaultServlet (same as in HTTPDemon.parseMultipart) - reduce Jetty logging - give build.run a bit more memory (set to YaCy.default 600m from 512m) Changed Files: build.xml, defaults/yacy.logging, source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Sun Oct 13 18:06:42 CEST 2013 by reger | - add localpeername.yacy to list of local address detection for AbstractRemoteHandler - use proxy via header info as in legacy proxy handler Changed Files: source/net/yacy/http/AbstractRemoteHandler.java, source/net/yacy/http/ProxyHandler.java |
Sun Oct 13 17:56:53 CEST 2013 by reger | fix YaCy version string in proxy headers (config parameter vString not longer used) Changed Files: source/net/yacy/server/http/HTTPDProxyHandler.java |
Sun Oct 13 07:24:33 CEST 2013 by reger | disable domainhandler due to error - domainhandler causes closed response output stream in following handlers on addresses resolved to local peer (like in hello protocoll preventing peer to switch to senior peer) Changed Files: source/net/yacy/http/Jetty8HttpServerImpl.java |
Sat Oct 12 23:01:14 CEST 2013 by reger | update Info.plist small DefaultServlet refactoring Changed Files: addon/YaCy.app/Contents/Info.plist, htroot/Connections_p.java, source/net/yacy/http/Jetty8YaCyDefaultServlet.java, source/net/yacy/http/YaCyDefaultServlet.java |
Thu Oct 10 23:23:58 CEST 2013 by reger | searchpage javascript (latestinfo) causes reset of search statistic after moving to next page - disabled call via setTimeout in yacysearch.html Changed Files: htroot/yacysearch.html, source/net/yacy/http/Jetty8YaCyDefaultServlet.java |
Thu Oct 10 20:57:37 CEST 2013 by sixcooler | fix for ssl-connections from proxy-usage staying in close-wait-state + some extra 'close' in HttpClient Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/server/http/HTTPDProxyHandler.java |
Wed Oct 09 22:07:59 CEST 2013 by Michael Peter Christen | added new icon/image for latest commit Changed Files: htroot/env/grafics/right.gif |
Wed Oct 09 22:07:32 CEST 2013 by Michael Peter Christen | - replaced 'edit' link with a clone symbol in Table_API_p since that is what it does: it clones the crawl, it does not change the crawl. - moved the appearance of this clone link to the type column since this makes it visible also if the URL column is not visible. Changed Files: htroot/Table_API_p.html, htroot/Table_API_p.java, htroot/env/grafics/doc.gif |
Wed Oct 09 21:32:27 CEST 2013 by Michael Peter Christen | Merge remote-tracking branch 'jensbees/crawlexpert-post' Changed Files: htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java, htroot/Table_API_p.html, htroot/Table_API_p.java |
Wed Oct 09 21:06:11 CEST 2013 by bhoerdzn | rolling back wrong merge Changed Files: |
Wed Oct 09 14:23:06 CEST 2013 by bhoerdzn | reverting local changes to project.xml Changed Files: nbproject/project.xml |
Wed Oct 09 13:52:44 CEST 2013 by bhoerdzn | check URL on inital load, if set Changed Files: htroot/CrawlStartExpert_p.html |
Wed Oct 09 13:38:20 CEST 2013 by bhoerdzn | add a hidden field for "crawlingstart" since jQuery omits the submit button value Changed Files: htroot/CrawlStartExpert_p.html |
Wed Oct 09 12:55:42 CEST 2013 by bhoerdzn | link recorded crawl starts back to "CrawlStartExpert_p" in "Process Scheduler" Changed Files: htroot/Table_API_p.html, htroot/Table_API_p.java |
Wed Oct 09 12:42:04 CEST 2013 by bhoerdzn | use integer types for boolean values Changed Files: htroot/CrawlStartExpert_p.java |
Wed Oct 09 10:42:35 CEST 2013 by bhoerdzn | correct state handling for country code restriction Changed Files: htroot/CrawlStartExpert_p.html |
Wed Oct 09 10:36:08 CEST 2013 by bhoerdzn | correctly set initial state for load filters Changed Files: htroot/CrawlStartExpert_p.html |
Wed Oct 09 10:05:51 CEST 2013 by bhoerdzn | set form boolean values to 0 & 1 instead of false & true Changed Files: htroot/CrawlStartExpert_p.html |
Wed Oct 09 09:48:00 CEST 2013 by bhoerdzn | Remove some post parameters, if they are set to default values, as their values are already set by YaCy. Added some documentation. Changed Files: htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java |
Wed Oct 09 01:07:22 CEST 2013 by reger | make legacy http server (serverCore) implement YaCyHttpServer interface Changed Files: source/net/yacy/server/serverCore.java |
Tue Oct 08 23:48:13 CEST 2013 by Michael Peter Christen | - Added new solr field url_file_name_tokens_t which stores the file name tokens. This can be used to enhance the ranking. - Added also a rating_i field as basis for later usage. - enhanced the tokenization process. Changed Files: defaults/solr.collection.schema, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java |
Tue Oct 08 18:41:07 CEST 2013 by orbiter | added the anchor text within web pages to the searcheable entities of a web page. This can be of benefit for the ranking if these fields are used for boosts. Changed Files: defaults/solr.collection.schema, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Tue Oct 08 18:15:35 CEST 2013 by orbiter | list more fields available for search and for ranking boosts Changed Files: htroot/RankingSolr_p.html, htroot/RankingSolr_p.java, source/net/yacy/search/schema/CollectionSchema.java |
Tue Oct 08 18:02:42 CEST 2013 by sixcooler | fix false fill NAME_CACHE_MISS-DNS-Cache in case of a timeout for eg. caused by massive requests when crawl from file Changed Files: source/net/yacy/cora/protocol/Domains.java |
Tue Oct 08 13:26:09 CEST 2013 by bhoerdzn | Use list template for "reloadIfOlderNumber". Changed Files: htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java |
Tue Oct 08 13:24:27 CEST 2013 by bhoerdzn | Dynamic toggling of form fields, based on passed in and selected values. This will also cut down the post string by disabling not needed fields. Changed Files: htroot/CrawlStartExpert_p.html |
Mon Oct 07 23:51:28 CEST 2013 by Michael Peter Christen | removed unused initialization method Changed Files: source/net/yacy/search/query/QueryParams.java |
Mon Oct 07 23:41:50 CEST 2013 by Michael Peter Christen | removed warnings Changed Files: htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, htroot/yacysearch.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Mon Oct 07 22:02:42 CEST 2013 by Lotus | adding synonyms from German OpenThesaurus ready for use in YaCy Changed Files: addon/synonyms/LICENSE.txt, addon/synonyms/openthesaurus_de_yacy |
Mon Oct 07 13:13:22 CEST 2013 by bhoerdzn | try to guess default crawling mode, if none set Changed Files: htroot/CrawlStartExpert_p.java |
Mon Oct 07 13:01:39 CEST 2013 by bhoerdzn | assign strings before checking contents Changed Files: htroot/CrawlStartExpert_p.java |
Mon Oct 07 12:32:23 CEST 2013 by bhoerdzn | set default values for string parameters Changed Files: htroot/CrawlStartExpert_p.java |
Mon Oct 07 11:25:59 CEST 2013 by bhoerdzn | made crawStartExpert_p aware of post variables; extended template where needed Changed Files: htroot/CrawlStartExpert_p.html, htroot/CrawlStartExpert_p.java, nbproject/project.xml |
Mon Oct 07 11:15:58 CEST 2013 by Michael Peter Christen | Patch the citation index for links with canonical tags. This shall fulfill the following requirement: If a document A links to B and B contains a 'canonical C', then the citation rank computation shall consider that A links to C and B does not link to C. To do so, we first must collect all canonical links, find all references to them, get the anchor list of the documents and patch the citation reference of these links. Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Fri Oct 04 01:16:17 CEST 2013 by reger | add call to AccessTracker to jetty security handler Changed Files: source/net/yacy/http/YaCySecurityHandler.java |
Thu Oct 03 19:38:03 CEST 2013 by reger | adjust YaCySecurityHandler to Jetty 9 conventions - mainly adjust prepareConstraintInfo to use the RoleInfo.setChecked as in Jetty Source distribution - use constraint check behavior as in ConstraintSecurityHandler see http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/jetty-security/src/main/java/org/eclipse/jetty/security/ConstraintSecurityHandler.java?id=jetty-9.0.5.v20130813 Changed Files: source/net/yacy/http/YaCySecurityHandler.java |
Tue Oct 01 03:04:32 CEST 2013 by reger | - expand localHostName check of AbstractRemoteHandler to pevent request is handled as proxy request - make domain handler not relay on included path in resolved .yacy address Changed Files: source/net/yacy/http/AbstractRemoteHandler.java, source/net/yacy/http/YacyDomainHandler.java |
Mon Sep 30 03:47:55 CEST 2013 by reger | upd build script from rc1/master Changed Files: build.xml |
Mon Sep 30 03:12:52 CEST 2013 by reger | update Jetty server init() to activate yacy-domain and transparent proxy handler - adding domain & proxy handler to a context (as it was in inital design) (context required for dispatcher) - make handler context and servlet context parallel available (to allow use of YaCyDefaultServlet to handle legacyServlets) - set transparent proxy request handled after dispatch.forward to skip further handling for .yacy domain requests Changed Files: source/net/yacy/http/HttpServer.java, source/net/yacy/http/YacyDomainHandler.java |
Sun Sep 29 03:34:00 CEST 2013 by reger | - add welcome file (index.html) support to YaCyDefaultServlet - change SolrServlet default search field (&df) to text_t Changed Files: source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/http/HttpServer.java, source/net/yacy/http/YaCyDefaultServlet.java |
Fri Sep 27 16:57:05 CEST 2013 by Michael Peter Christen | added two more fields source_cr_host_norm_i,target_cr_host_norm_i in webgraph and an addition to postprocessing to copy all cr ranking attributes to the link edges associated to the postprocessing documents Changed Files: defaults/solr.webgraph.schema, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/WebgraphSchema.java |
Fri Sep 27 13:17:50 CEST 2013 by Michael Peter Christen | added parsing of http-canonical tags (untested, could not find an example page) Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Fri Sep 27 12:28:09 CEST 2013 by Michael Peter Christen | do not fail if a wrong boost key is used; instead, print only a warning See also: http://bugs.yacy.net/view.php?id=293 Changed Files: source/net/yacy/cora/federate/solr/Ranking.java |
Thu Sep 26 22:20:35 CEST 2013 by reger | add default field (&df) to SolrServlet query if missing Changed Files: source/net/yacy/cora/federate/solr/SolrServlet.java, source/net/yacy/http/HttpServer.java |
Thu Sep 26 10:22:31 CEST 2013 by Michael Peter Christen | - crawl profiles shall be deleted from active and passive stacks if they are deleted to terminate the crawl because otherwise the crawl will go on after the load-from-passive stack policy. - better check if a crawl is terminated using the loader queue. Changed Files: htroot/CrawlProfileEditor_p.java, htroot/Crawler_p.java, htroot/IndexCreateQueues_p.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/search/Switchboard.java, source/net/yacy/server/serverSwitch.java |
Wed Sep 25 21:14:52 CEST 2013 by Michael Peter Christen | hack to remove most of the warning: deprecated messages (but not all, one is left) Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Wed Sep 25 18:45:16 CEST 2013 by sixcooler | reverted RemoteInstance to deprecated methods of httpClient-4.2 this should work with current remote-Solr-Instances Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Wed Sep 25 15:01:28 CEST 2013 by sixcooler | some dns-timing changes: since httpclient uses the domain-cache it is useful not to clean the domain cache until crawling is running (domains are filled into this cache) On huge crawl-starts (eg. from file) my DNS did not follow the high rates - so I reduced the rate and give some more time(-out) Changed Files: source/net/yacy/cora/protocol/Domains.java, source/net/yacy/search/Switchboard.java |
Wed Sep 25 11:04:12 CEST 2013 by orbiter | when profiles are cleaned, there shall be first a callback showing which profiles are cleaned. This shall enable a profile-termination-driven postprocessing. To do this, index writings must carry the profile key which will be implemented in another (next) step. Changed Files: source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/search/Switchboard.java |
Tue Sep 24 21:18:37 CEST 2013 by orbiter | removed superfluous class Changed Files: |
Tue Sep 24 21:08:40 CEST 2013 by orbiter | Added new data structure to be used by the balancer (not used yet). These data structures will enable the balancer to store the crawl queue into individual queues, one each for a single host. Changed Files: source/net/yacy/crawler/HostQueue.java, source/net/yacy/crawler/HostQueues.java |
Tue Sep 24 19:55:59 CEST 2013 by orbiter | refactoring Changed Files: source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/data/Latency.java |
Tue Sep 24 11:26:51 CEST 2013 by orbiter | added location navigator which causes that the image to the map search is visible whenever a location is available in the search result. To activate this, the search.navigation property in yacy.conf must be modified to the new default values. Changed Files: defaults/yacy.init, htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, htroot/yacysearch.java, htroot/yacysearchtrailer.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Tue Sep 24 11:23:45 CEST 2013 by orbiter | automatically removed Places autotagging if no location library is wanted Changed Files: htroot/DictionaryLoader_p.java, source/net/yacy/cora/language/synonyms/AutotaggingLibrary.java |
Mon Sep 23 18:09:42 CEST 2013 by Michael Peter Christen | - added hosthash to all documents (also fail documents which is needed there for deletion), this fixes a problem for the deletion of old documents for new crawl starts - added clickdepth and citation computation for fail documents Changed Files: source/net/yacy/search/index/Segment.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Mon Sep 23 12:19:51 CEST 2013 by Michael Peter Christen | close include byte buffer after usage Changed Files: source/net/yacy/server/http/TemplateEngine.java |
Mon Sep 23 03:01:08 CEST 2013 by reger | remove double declaration of TLD_any_zone_filter Changed Files: source/net/yacy/cora/document/id/DigestURL.java |
Mon Sep 23 01:30:45 CEST 2013 by reger | - reduce Jetty debug logging - fix Context path initialization Changed Files: defaults/yacy.logging, source/net/yacy/http/HttpServer.java |
Sun Sep 22 20:57:50 CEST 2013 by reger | adjust to DigestURI changes from master to DigestURL Changed Files: source/net/yacy/http/ProxyCacheHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/YaCySecurityHandler.java |
Sun Sep 22 04:23:42 CEST 2013 by reger | use more YaCy HeaderFramework constants (instead of Jetty's) Changed Files: source/net/yacy/http/HttpServer.java, source/net/yacy/http/YaCyDefaultServlet.java |
Sun Sep 22 01:57:32 CEST 2013 by reger | implement a YaCyDefaultServlet to handle YaCy-servlets within Jetty server - the implementation is inspired by Jetty's DefaultServlet - handles static html content and YaCy servlets - translates between standard servlet request/response and YaCy request/response specification With the implementation of YaCy-servlets as servlet instead via a jetty handler it's closer to servlet standard and carries less jetty specific dependencies. Changed Files: source/net/yacy/http/CrashProtectionHandler.java, source/net/yacy/http/HttpServer.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/YaCyDefaultServlet.java |
Sat Sep 21 08:57:43 CEST 2013 by orbiter | hack to add all tokens from the url to text_t. This was working for the RWI index (and still is working) but not for solr-only search indexes. Maybe we should find a solution using a separate search field instead. Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Sat Sep 21 08:56:24 CEST 2013 by orbiter | do not check all words with synonym library if the library is empty Changed Files: source/net/yacy/cora/language/synonyms/SynonymLibrary.java, source/net/yacy/document/Condenser.java |
Fri Sep 20 16:52:37 CEST 2013 by orbiter | disabled ipv4 preference to enable ipv6-only networks like freifunk Changed Files: source/net/yacy/yacy.java |
Fri Sep 20 16:51:50 CEST 2013 by orbiter | CPU problem when pusing to the error cache; wrong class, ConcurrentHashMap needed for concurrency Changed Files: source/net/yacy/search/index/ErrorCache.java |
Tue Sep 17 15:52:57 CEST 2013 by Michael Peter Christen | better and consistent deletions for error urls Changed Files: htroot/Crawler_p.java, htroot/QuickCrawlLink_p.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/ErrorCache.java |
Mon Sep 16 16:14:56 CEST 2013 by Michael Peter Christen | set anchor rel attribute of all links to "nofollow" if the html meta contains a robots:nofollow or if the http header contains a "X-Robots-Tag: nofollow" Changed Files: source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Mon Sep 16 03:01:18 CEST 2013 by reger | add Solr Servlet Changed Files: source/net/yacy/http/HttpServer.java |
Sat Sep 14 21:10:24 CEST 2013 by reger | Merge remote-tracking branch 'origin/master' into jetty Changed Files: htroot/gsa/searchresult.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/crawler/data/Cache.java |
Sat Sep 14 20:56:46 CEST 2013 by reger | update Info.plist for Jetty 9 jars Changed Files: addon/YaCy.app/Contents/Info.plist |
Sat Sep 14 18:29:44 CEST 2013 by reger | remove redundant declaration of USER_AGENT Changed Files: source/net/yacy/cora/protocol/RequestHeader.java |
Thu Sep 12 22:49:09 CEST 2013 by reger | add mandatory p2p parameter to templatePattern Changed Files: source/net/yacy/http/TemplateHandler.java |
Thu Sep 12 22:04:10 CEST 2013 by reger | adjust setHandled to request of call parameter Changed Files: source/net/yacy/http/TemplateHandler.java |
Thu Sep 12 22:02:11 CEST 2013 by reger | remove references to 10_http thread (legacy http server) and add needed get/set function to jetty http server wrapper Changed Files: htroot/ConfigBasic.java, htroot/PerformanceQueues_p.java, htroot/SettingsAck_p.java, htroot/Status.java, source/net/yacy/http/HttpServer.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/yacy.java |
Wed Sep 11 13:03:58 CEST 2013 by Michael Peter Christen | decreased the responseHeaderDB database which is now flushed more frequently. This will preserve more documents in the cache in case of a crash. Changed Files: source/net/yacy/crawler/data/Cache.java |
Wed Sep 11 12:52:18 CEST 2013 by Michael Peter Christen | added option for daterange properties in GSA interface to use an left- or right-open date range; i.e. using daterange=..2013-09-09 or daterange=2013-09-02.. additional to daterange=2013-09-02..2013-09-09 Changed Files: htroot/gsa/searchresult.java |
Wed Sep 11 02:24:47 CEST 2013 by reger | - remove double initialization of jetty - refactor some var assignments Changed Files: htroot/Connections_p.java, source/net/yacy/http/HttpServer.java, source/net/yacy/http/SSIHandler.java, source/net/yacy/http/TemplateHandler.java |
Tue Sep 10 20:55:03 CEST 2013 by reger | - delete not needed old jetty jars from libt - add jetty to Info.plist Changed Files: addon/YaCy.app/Contents/Info.plist |
Tue Sep 10 20:05:10 CEST 2013 by reger | adjust test/ParserTest to dc_description data type Changed Files: test/de/anomic/document/ParserTest.java |
Mon Sep 09 12:58:26 CEST 2013 by Michael Peter Christen | added tools to visualize image generation (i.e. during testing) Changed Files: source/net/yacy/search/Switchboard.java, source/net/yacy/visualization/AnimationPlotter.java, source/net/yacy/visualization/HexGridPlotter.java, source/net/yacy/visualization/RasterPlotter.java |
Mon Sep 09 02:37:29 CEST 2013 by reger | changes to adjust jetty to recent code changes Changed Files: .classpath, source/net/yacy/http/ContentModHandler.java, source/net/yacy/http/ProxyCacheHandler.java, source/net/yacy/http/ProxyHandler.java, source/net/yacy/http/SSIHandler.java, source/net/yacy/http/TemplateHandler.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YacyDomainHandler.java, source/net/yacy/yacy.java |
Thu Sep 05 13:22:16 CEST 2013 by Michael Peter Christen | forced deletion of ZURL entries for a specific host for each host that appears in the crawl url list Changed Files: htroot/Crawler_p.java, source/net/yacy/cora/document/ASCII.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/search/Switchboard.java |
Thu Sep 05 09:59:41 CEST 2013 by Michael Peter Christen | refactoring (im preparation for new removeHost method) Changed Files: htroot/HostBrowser.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/search/Switchboard.java |
Wed Sep 04 16:00:47 CEST 2013 by orbiter | fix for a problem which ocurres when a site is crawled where the start url is redirected. Changed Files: source/net/yacy/crawler/data/ZURL.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/search/Switchboard.java |
Wed Sep 04 10:47:18 CEST 2013 by Michael Peter Christen | turned images_alt_txt back to images_alt_sxt because it is not necessary to index the alt text. Indexed image Text is in images_text_t Changed Files: defaults/solr.collection.schema, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java |
Tue Sep 03 13:34:01 CEST 2013 by Michael Peter Christen | activated image type navigation for image search Changed Files: source/net/yacy/search/query/SearchEvent.java |
Tue Sep 03 11:14:23 CEST 2013 by Michael Peter Christen | added an option to exclude image search results from text search. This is on by default. Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Tue Sep 03 11:13:45 CEST 2013 by Michael Peter Christen | added an option to put image links to the crawl queue and handle these like normal documents. Using this option (by default on at this moment; this might change soon) it is possible to get the exif data into the search index to be used in image search. Changed Files: defaults/yacy.init, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/Switchboard.java |
Tue Sep 03 09:48:05 CEST 2013 by Michael Peter Christen | avoid to create unnecessary objects Changed Files: source/net/yacy/search/query/SearchEvent.java |
Tue Sep 03 09:47:27 CEST 2013 by Michael Peter Christen | moving image description text to image text field Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Mon Sep 02 23:09:43 CEST 2013 by reger | remove all_words, all_strings lists from QueryGoal - only used for text highlighting in parser text (ViewFile.html) which can be done with include_strings only Changed Files: htroot/yacysearchitem.java, source/net/yacy/search/query/QueryGoal.java |
Mon Sep 02 04:19:53 CEST 2013 by reger | optimized QueryGoal (reducing mem and computation by removing all_hashes) - all_hashes used for text highlighting and word distance computation which can be done with include_hashes only Changed Files: htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, htroot/yacysearch.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/snippet/TextSnippet.java |
Sun Sep 01 14:35:36 CEST 2013 by orbiter | link strings should not be tokenized Changed Files: defaults/solr.collection.schema, htroot/HostBrowser.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java |
Sun Sep 01 13:48:10 CEST 2013 by orbiter | image alt tag strings should be tokenized Changed Files: defaults/solr.collection.schema, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java |
Sun Sep 01 13:12:36 CEST 2013 by orbiter | next development release step to reflect the extension of the solr api with javabin format capability Changed Files: build.properties |
Sun Sep 01 13:11:40 CEST 2013 by orbiter | added special handling of the BinaryResponseWriter in the solr interface which makes it possible to use solrj with the javabin format which is much better (compressed, no xml overhead, java object streams) and faster. Furthermore, this enables the 'shards' option in the solr interface which connects one solr (YaCy) to another solr (YaCy) ad-hoc. Changed Files: htroot/solr/select.java |
Sat Aug 31 06:25:12 CEST 2013 by reger | return field names in index instead of in schema for SolrServerConnector.getFields Changed Files: source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java |
Mon Aug 26 12:49:39 CEST 2013 by Michael Peter Christen | index migration to lucene 4.4 Changed Files: defaults/solr/solrconfig.xml, source/net/yacy/search/index/Fulltext.java |
Thu Aug 22 15:56:09 CEST 2013 by Michael Peter Christen | fixed a crawler bug where a double-occurring url was not re-crawled because the double-check error was written to the error-db and never deleted. No the error-db is cleared on every start and these double-messages are not written to the error-db any more. Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlQueues.java |
Thu Aug 22 10:12:25 CEST 2013 by Michael Peter Christen | removed an assert which causes a shortcut call circuit Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataRow.java |
Tue Aug 20 16:31:34 CEST 2013 by Michael Peter Christen | added missing files :( Changed Files: source/net/yacy/cora/util/ByteArray.java, source/net/yacy/cora/util/ByteBuffer.java |
Mon Aug 19 17:53:37 CEST 2013 by reger | add back menu to Surftips page (currently no menu is displayed) Changed Files: htroot/Surftips.html |
Mon Aug 19 10:54:23 CEST 2013 by Michael Peter Christen | removed link.gif as link button because this image is now shown automatically for expernal links Changed Files: htroot/HostBrowser.html |
Sat Aug 17 09:25:32 CEST 2013 by Michael Peter Christen | less logging for postprocessing (this was a debugging logging with high CPU load) Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Thu Aug 15 00:15:01 CEST 2013 by reger | revert del _blank (last commit) in template Changed Files: htroot/env/templates/header.template |
Thu Aug 15 00:05:32 CEST 2013 by reger | add some href target=_blank to ext. links with external icon Changed Files: htroot/CrawlResults.html, htroot/IndexCreateQueues_p.html, htroot/Vocabulary_p.html, htroot/env/templates/header.template |
Wed Aug 14 21:12:10 CEST 2013 by reger | - remove possible double initialization of rdfa parser - use ordered list to use preferred parser for mime/extension first (relates to html, rdfa, argument parser) - harmonize xhtml extension config for the 3 html base parsers Changed Files: source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java |
Wed Aug 14 00:00:30 CEST 2013 by reger | FieldReIndex: omit active vocabulary fields from reindex detection Changed Files: source/net/yacy/migration.java |
Tue Aug 13 03:07:25 CEST 2013 by reger | make Vocabulary Navigator tags a hard result entry filter by checking vocabulary tags also for rwi results (currently a filter is applied to the solr query) TODO: as vocabularies are only locally valid, auto-switch to Searchdom.LOCAL could be considered. Changed Files: source/net/yacy/search/query/SearchEvent.java |
Mon Aug 12 21:20:23 CEST 2013 by reger | improve tld: query modifier filter pattern (to prevent tld:net accepting www.abcinet.org) Changed Files: source/net/yacy/search/query/QueryParams.java |
Sun Aug 11 04:51:29 CEST 2013 by reger | Field Re-Indexing: display list of fields in reindex queue change servlet to display statistic on 1st click (instead after refresh) Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, source/net/yacy/search/index/ReindexSolrBusyThread.java |
Thu Aug 08 14:38:26 CEST 2013 by sixcooler | clear some caches before reporting low Memory do not break lines in Network-table-rows Changed Files: htroot/Network.html, source/net/yacy/search/ResourceObserver.java |
Thu Aug 08 00:55:21 CEST 2013 by reger | Index Administration - Field Re-Indexing: exclude internal Solr _version_ field from obsolete field check Changed Files: source/net/yacy/migration.java |
Tue Aug 06 03:58:53 CEST 2013 by sixcooler | rise autoCommit maxTime to 3 Minutes to reduce IO lower mergeFactor again (5) for less segments Changed Files: defaults/solr/solrconfig.xml |
Tue Aug 06 03:48:29 CEST 2013 by sixcooler | add branch to packet-name if not build from master Changed Files: build.properties, build.xml, libbuild/GitRevTask/GitRevTask.java |
Thu Aug 01 16:40:58 CEST 2013 by sixcooler | fix / workaround for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4750 + Seed.hash should be final Changed Files: htroot/yacy/hello.java, source/net/yacy/peers/Seed.java |
Thu Aug 01 12:35:44 CEST 2013 by Michael Peter Christen | protection against self-ping (may be cause by fraud attempts) Changed Files: htroot/yacy/hello.java |
Thu Aug 01 12:11:56 CEST 2013 by orbiter | added donation plea to the about box as default (you can replace this in your peer!) Changed Files: defaults/yacy.init |
Thu Aug 01 11:49:07 CEST 2013 by orbiter | more space in the about section Changed Files: htroot/yacysearchtrailer.html |
Wed Jul 31 13:32:05 CEST 2013 by Michael Peter Christen | fix to collection schema; bug appeared for _txt fields with empty String as content Changed Files: source/net/yacy/search/schema/CollectionSchema.java |
Wed Jul 31 13:16:32 CEST 2013 by Michael Peter Christen | added a stub of a class for crawler redesign Changed Files: source/net/yacy/crawler/CrawlQueue.java |
Tue Jul 30 21:40:37 CEST 2013 by Marc Nause | Improved external links. *) image links will not be marked (if they have class "yacylogo" or "forceNoExternalIcon") *) external links in menu on left (and "fork me"-banner) will open in new tab/window now Changed Files: htroot/Status.html, htroot/env/base.css, htroot/env/templates/header.template |
Tue Jul 30 20:46:51 CEST 2013 by Marc Nause | External links in HTML interface are marked as external with small icon. *) added new icon *) added CSS rules to mark all external links except search results (target="_self") Changed Files: htroot/env/base.css, htroot/env/grafics/external.png, htroot/index.html |
Tue Jul 30 14:32:59 CEST 2013 by Michael Peter Christen | fix for NPE which happens within solr code at MultiMapSolrParams.java, line 52 in case that the array arr.length == 0 Changed Files: source/net/yacy/server/serverObjects.java |
Tue Jul 30 14:14:16 CEST 2013 by Michael Peter Christen | update to pdfbox 1.8.2 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/fontbox-1.8.2.License, lib/fontbox-1.8.2.jar, lib/jempbox-1.8.2.License, lib/jempbox-1.8.2.jar, lib/pdfbox-1.8.2.License, lib/pdfbox-1.8.2.jar |
Tue Jul 30 13:05:32 CEST 2013 by Michael Peter Christen | cleaned sourcepaths from eclipse classpath Changed Files: .classpath |
Tue Jul 30 12:51:00 CEST 2013 by Michael Peter Christen | next development cycle Changed Files: build.properties |
Mon Jul 29 01:32:02 CEST 2013 by reger | Field Re-Indexing: prevent endless error loop in ReindexSolrBusyThread on Solr exception (by skipping query causing the exception) (occured during testing while working on q=store:[* TO *]) Changed Files: source/net/yacy/search/index/ReindexSolrBusyThread.java |
Sun Jul 28 03:41:09 CEST 2013 by reger | htmlParser: closes input stream -> changed it to leave it open for a reset (used by AugmentParser - even if this is practically not used), note: stream.close is done by caller (Textparser.parseSource) - removed unnecessary reset in AugmentParser - added stream.mark in tdfatripleimpl. to make stream.reset work here Changed Files: source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaTripleImpl.java |
Sat Jul 27 15:36:46 CEST 2013 by orbiter | reverted start script options - yacy on windows did not start with the given values Changed Files: addon/YaCy.app/Contents/Info.plist, addon/yacyInit.m4, startYACY.bat, startYACY.sh, startYACY_debug.bat |
Sat Jul 27 11:59:07 CEST 2013 by orbiter | emergency fix for crawler: this will otherwise cause loss of complete crawl queue if latency of remote system is too low Changed Files: source/net/yacy/crawler/Balancer.java |