Release 1.1 Most important about this release is the replacement of the built-in metadata database with apache solr. Solr is an indexing framework of its own ant the current implementation uses only a fragemt of the functionalities that is provided by solr but the embedded solr server is accessible from outside as a http service as a jetty- or apache http-driven version of solr does. This interface is now also used by YaCy using so-called new 'Node' peers which are highly available and very fast peers in the YaCy p2p network. The solr Schema has a rich number of metadata fields, where the original number of metadata field of YaCy is a subset. YaCy 1.1 is much faster and presents more and more appropriate search results when doing a more-than-one-word search.
Commit | Description |
---|---|
Mon Aug 20 12:16:11 CEST 2012 by Michael Peter Christen | added remote search to solr on YaCy peers! - when doing a remote search, node peers are selected for solr queries - the solr query is done concurrently to the standard YaCy rwi search - the solr search result is feeded into the same data structure that prepares the rwi search result - the same remote seach that is done to several outside peers is done to the local solr index - the search process works now also without any 'old' RWI data using solr Changed Files: .classpath, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java |
Fri Aug 17 15:52:33 CEST 2012 by Michael Peter Christen | refactoring Changed Files: htroot/Bookmarks.java, htroot/CrawlResults.java, htroot/Crawler_p.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexFederated_p.java, htroot/IndexShare_p.java, htroot/QuickCrawlLink_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/YBRFetch_p.java, htroot/api/schema_p.java, htroot/api/status_p.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/gsa/searchresult.java, htroot/solr/select.java, htroot/yacy/crawlReceipt.java, htroot/yacy/query.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/de/anomic/crawler/CrawlQueues.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/ResultEntry.java |
Fri Aug 17 14:43:32 CEST 2012 by Michael Peter Christen | code simplification Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, htroot/api/schema_p.java, htroot/gsa/searchresult.java, htroot/solr/select.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java |
Fri Aug 17 01:34:38 CEST 2012 by Michael Peter Christen | - refactoring (load -> getMetadata) - added getDocument to retrieve Solr documents which shall replace getMetadata Changed Files: htroot/Bookmarks.java, htroot/CrawlResults.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java |
Thu Aug 16 18:17:47 CEST 2012 by Michael Peter Christen | FOR THE BRAVE.. this is a forced migration to solr which is now ready for production as a replacement of the metadata-db. This intermediate release 1.041 will switch on the previously optional solr index and the old metadata-db will still work as it did before. Solr+metadata are accessed in mixed mode, no migration is done yet. If this causes not a catastrophe until the end of the weekend, we will do a YaCy 1.1 main release containing this as default. Changed Files: build.properties, defaults/yacy.init, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Thu Aug 16 16:28:57 CEST 2012 by Michael Peter Christen | get the peer location more quickly Changed Files: htroot/Network.java, htroot/gsa/searchresult.java, htroot/solr/select.java, htroot/yacy/hello.java, htroot/yacy/query.java, htroot/yacy/transferRWI.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/dht/PeerSelection.java |
Mon Aug 13 16:48:14 CEST 2012 by cominch | Merge remote-tracking branch 'original yacy/master' Changed Files: .classpath, .settings/org.eclipse.jdt.core.prefs, addon/YaCy.app/Contents/Info.plist, bin/localsearch.sh, build.properties, build.xml, defaults/httpd.mime, defaults/solr.keys.list, defaults/solr/schema.xml, defaults/solr/solrconfig.xml, defaults/yacy.init, defaults/yacy.network.freeworld.unit, htroot/AccessGrid_p.java, htroot/AccessPicture_p.java, htroot/AccessTracker_p.java, htroot/AugmentedBrowsingFilters_p.java, htroot/AugmentedBrowsing_p.java, htroot/AugmentedParsing_p.java, htroot/Banner.java, htroot/BlacklistCleaner_p.java, htroot/BlacklistImpExp_p.java, htroot/BlacklistTest_p.java, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/CacheResource_p.java, htroot/ConfigAccounts_p.java, htroot/ConfigAppearance_p.java, htroot/ConfigBasic.java, htroot/ConfigHTCache_p.java, htroot/ConfigHeuristics_p.html, htroot/ConfigHeuristics_p.java, htroot/ConfigLanguage_p.java, htroot/ConfigLiveSearch.java, htroot/ConfigNetwork_p.java, htroot/ConfigPortal.java, htroot/ConfigProfile_p.java, htroot/ConfigProperties_p.java, htroot/ConfigRobotsTxt_p.java, htroot/ConfigSearchBox.java, htroot/ConfigUpdate_p.java, htroot/Connections_p.java, htroot/ContentIntegrationPHPBB3_p.java, htroot/CookieMonitorIncoming_p.java, htroot/CookieMonitorOutgoing_p.java, htroot/CrawlMonitorRemoteStart.java, htroot/CrawlProfileEditor_p.java, htroot/CrawlResults.java, htroot/CrawlStartExpert_p.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/DemoServlet.java, htroot/DemoServletInteraction.java, htroot/DemoServletRDF.java, htroot/DictionaryLoader_p.java, htroot/Help.java, htroot/IndexControlRWIs_p.html, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/IndexCreateDomainCrawl_p.java, htroot/IndexCreateLoaderQueue_p.java, htroot/IndexCreateParserErrors_p.java, htroot/IndexCreateQueues_p.java, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/IndexImportMediawiki_p.java, htroot/IndexImportOAIPMHList_p.java, htroot/IndexImportOAIPMH_p.java, htroot/IndexShare_p.java, htroot/Load_MediawikiWiki.java, htroot/Load_PHPBB3.java, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Network.html, htroot/Network.java, htroot/NetworkPicture.java, htroot/News.java, htroot/PeerLoadPicture.java, htroot/PerformanceConcurrency_p.java, htroot/PerformanceGraph.java, htroot/PerformanceMemory_p.java, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, htroot/PerformanceSearch_p.java, htroot/ProxyIndexingMonitor_p.java, htroot/QuickCrawlLink_p.java, htroot/Ranking_p.java, htroot/RemoteCrawl_p.java, htroot/SearchEventPicture.java, htroot/ServerScannerList.java, htroot/SettingsAck_p.java, htroot/Status.java, htroot/Supporter.java, htroot/Surftips.java, htroot/Table_API_p.java, htroot/Table_RobotsTxt_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.java, htroot/Threaddump_p.java, htroot/Trails.java, htroot/Triple_p.java, htroot/Triplestore_p.java, htroot/User.java, htroot/ViewFile.html, htroot/ViewFile.java, htroot/ViewImage.java, htroot/ViewLog_p.java, htroot/Vocabulary_p.java, htroot/WatchWebStructure_p.java, htroot/WebStructurePicture_p.java, htroot/WikiHelp.java, htroot/YBRFetch_p.java, htroot/YMarks.java, htroot/api/blacklists.java, htroot/api/blacklists_p.java, htroot/api/bookmarks/xbel/xbel.java, htroot/api/config_p.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/latency_p.java, htroot/api/schema_p.java, htroot/api/status_p.java, htroot/api/table_p.java, htroot/api/termlist_p.java, htroot/api/timeline.java, htroot/api/trail_p.java, htroot/api/version.java, htroot/api/webstructure.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/delete_ymark.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/import_ymark.java, htroot/api/ymarks/manage_tags.java, htroot/api/ynetSearch.java, htroot/compare_yacy.java, htroot/env/style.java, htroot/env/templates/submenuBlacklist.template, htroot/imagetest.java, htroot/interaction/GetRDF.java, htroot/interaction/PutRDF.java, htroot/interaction/Table.java, htroot/interaction_elements/Document_part.java, htroot/interaction_elements/Footer.java, htroot/interaction_elements/Loginstatus_part.java, htroot/interaction_elements/OverlayInteraction.java, htroot/interaction_elements/Tag_part.java, htroot/mediawiki_p.java, htroot/opensearchdescription.java, htroot/osm.java, htroot/rct_p.java, htroot/robots.java, htroot/sharedBlacklist_p.java, htroot/solr/select.java, htroot/ssitestservlet.java, htroot/suggest.java, htroot/test.java, htroot/www/welcome.java, htroot/yacy/crawlReceipt.java, htroot/yacy/hello.java, htroot/yacy/message.java, htroot/yacy/query.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacyinteractive.java, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, htroot/yacysearchtrailer.java, lib/dependencies.txt, lib/httpclient-4.2.1.License, lib/httpclient-4.2.1.jar, lib/httpcore-4.2.1.License, lib/httpcore-4.2.1.jar, lib/httpmime-4.2.1.License, lib/httpmime-4.2.1.jar, libbuild/GitRevTask/GitRevTask.java, libbuild/svnRevNr/svnRevNrParser.java, locales/cn.lng, locales/de.lng, locales/ru.lng, locales/uk.lng, nbproject/project.xml, source/de/anomic/crawler/Balancer.java, source/de/anomic/crawler/Cache.java, source/de/anomic/crawler/CrawlProfile.java, source/de/anomic/crawler/CrawlQueues.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/CrawlSwitchboard.java, source/de/anomic/crawler/NoticedURL.java, source/de/anomic/crawler/RSSLoader.java, source/de/anomic/crawler/ResultImages.java, source/de/anomic/crawler/ResultURLs.java, source/de/anomic/crawler/RobotsTxt.java, source/de/anomic/crawler/RobotsTxtEntry.java, source/de/anomic/crawler/RobotsTxtParser.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/crawler/ZURL.java, source/de/anomic/crawler/retrieval/FTPLoader.java, source/de/anomic/crawler/retrieval/FileLoader.java, source/de/anomic/crawler/retrieval/HTTPLoader.java, source/de/anomic/crawler/retrieval/Request.java, source/de/anomic/crawler/retrieval/Response.java, source/de/anomic/crawler/retrieval/SMBLoader.java, source/de/anomic/data/BlogBoard.java, source/de/anomic/data/BlogBoardComments.java, source/de/anomic/data/BookmarkDate.java, source/de/anomic/data/BookmarkHelper.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/data/DidYouMean.java, source/de/anomic/data/ListManager.java, source/de/anomic/data/MessageBoard.java, source/de/anomic/data/Translator.java, source/de/anomic/data/UserDB.java, source/de/anomic/data/WorkTables.java, source/de/anomic/data/wiki/AbstractWikiParser.java, source/de/anomic/data/wiki/WikiBoard.java, source/de/anomic/data/wiki/WikiCode.java, source/de/anomic/data/ymark/TablesRowComparator.java, source/de/anomic/data/ymark/YMarkAutoTagger.java, source/de/anomic/data/ymark/YMarkCrawlStart.java, source/de/anomic/data/ymark/YMarkDate.java, source/de/anomic/data/ymark/YMarkJSONImporter.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/de/anomic/data/ymark/YMarkTables.java, source/de/anomic/data/ymark/YMarkXBELImporter.java, source/de/anomic/http/server/AugmentedHtmlStream.java, source/de/anomic/http/server/ChunkedInputStream.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/http/server/HTTPDProxyHandler.java, source/de/anomic/http/server/HTTPDemon.java, source/de/anomic/http/server/ServerSideIncludes.java, source/de/anomic/http/server/TemplateEngine.java, source/de/anomic/server/serverCore.java, source/de/anomic/server/serverCoreSocket.java, source/de/anomic/server/serverObjects.java, source/de/anomic/server/serverSwitch.java, source/de/anomic/tools/CryptoLib.java, source/de/anomic/tools/UPnP.java, source/de/anomic/tools/crypt.java, source/de/anomic/tools/cryptbig.java, source/net/yacy/YaCySearchClient.java, source/net/yacy/ai/example/ConnectFour.java, source/net/yacy/ai/example/Hanoi.java, source/net/yacy/ai/example/SchwarzerPeter.java, source/net/yacy/ai/greedy/Battle.java, source/net/yacy/ai/greedy/Context.java, source/net/yacy/cora/date/GenericFormatter.java, source/net/yacy/cora/date/ISO8601Formatter.java, source/net/yacy/cora/document/ASCII.java, source/net/yacy/cora/document/Classification.java, source/net/yacy/cora/document/JSONObject.java, source/net/yacy/cora/document/JSONTokener.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/document/RSSReader.java, source/net/yacy/cora/language/phonetic/DoubleMetaphone.java, source/net/yacy/cora/language/phonetic/Metaphone.java, source/net/yacy/cora/language/phonetic/Soundex.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/Vocabulary.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/DCTerms.java, source/net/yacy/cora/lod/vocabulary/DublinCore.java, source/net/yacy/cora/lod/vocabulary/Foaf.java, source/net/yacy/cora/lod/vocabulary/Geo.java, source/net/yacy/cora/lod/vocabulary/HttpHeader.java, source/net/yacy/cora/lod/vocabulary/Owl.java, source/net/yacy/cora/lod/vocabulary/Rdf.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/lod/vocabulary/YaCyMetadata.java, source/net/yacy/cora/order/CloneableIterator.java, source/net/yacy/cora/order/CloneableMapIterator.java, source/net/yacy/cora/protocol/ByteArrayBody.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/services/federated/opensearch/SRURSSConnector.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/Schema.java, source/net/yacy/cora/services/federated/solr/ShardSelection.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SingleSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrDoc.java, source/net/yacy/cora/services/federated/solr/SolrType.java, source/net/yacy/cora/sorting/ClusteredScoreMap.java, source/net/yacy/cora/sorting/OrderedScoreMap.java, source/net/yacy/cora/storage/ConfigurationSet.java, source/net/yacy/cora/storage/Files.java, source/net/yacy/cora/storage/HandleMap.java, source/net/yacy/cora/storage/HandleSet.java, source/net/yacy/cora/storage/KeyList.java, source/net/yacy/cora/storage/SimpleARC.java, source/net/yacy/cora/storage/SizeLimitedMap.java, source/net/yacy/cora/storage/SizeLimitedSet.java, source/net/yacy/cora/util/SpaceExceededException.java, source/net/yacy/dbtest.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Autotagging.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/SentenceReader.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/StringBuilderComparator.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/WordCache.java, source/net/yacy/document/WordTokenizer.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/geolocation/GeoLocation.java, source/net/yacy/document/geolocation/GeonamesLocation.java, source/net/yacy/document/geolocation/Locations.java, source/net/yacy/document/geolocation/OpenGeoDBLocation.java, source/net/yacy/document/geolocation/OverarchingLocation.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/AbstractScraper.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ContentTransformer.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/bmpParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/gui/YaCyApp.java, source/net/yacy/gui/framework/Browser.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BLOB.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapColumnIndex.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/ObjectBuffer.java, source/net/yacy/kelondro/blob/Stack.java, source/net/yacy/kelondro/blob/Stacks.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/citation/CitationReference.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/MetadataVocabulary.java, source/net/yacy/kelondro/data/meta/URIMetadata.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/meta/URIReference.java, source/net/yacy/kelondro/data/meta/URIReferenceNode.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/data/word/WordReference.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/Cache.java, source/net/yacy/kelondro/index/Column.java, source/net/yacy/kelondro/index/Index.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/RAMIndex.java, source/net/yacy/kelondro/index/RAMIndexCluster.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/CachedFileReader.java, source/net/yacy/kelondro/io/CachedRecords.java, source/net/yacy/kelondro/io/CharBuffer.java, source/net/yacy/kelondro/io/RandomAccessIO.java, source/net/yacy/kelondro/io/Records.java, source/net/yacy/kelondro/logging/ConsoleOutHandler.java, source/net/yacy/kelondro/logging/GuiHandler.java, source/net/yacy/kelondro/logging/Log.java, source/net/yacy/kelondro/logging/LogParser.java, source/net/yacy/kelondro/logging/LogalizerHandler.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/kelondro/order/Base64Order.java, source/net/yacy/kelondro/order/Digest.java, source/net/yacy/kelondro/order/MergeIterator.java, source/net/yacy/kelondro/order/NaturalOrder.java, source/net/yacy/kelondro/order/RotateIterator.java, source/net/yacy/kelondro/order/StackIterator.java, source/net/yacy/kelondro/rwi/AbstractIndex.java, source/net/yacy/kelondro/rwi/AbstractReference.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/Index.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/rwi/IndexReader.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/rwi/TermSearch.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SQLTable.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/ByteArray.java, source/net/yacy/kelondro/util/ByteBuffer.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/GenerationMemoryStrategy.java, source/net/yacy/kelondro/util/MemoryControl.java, source/net/yacy/kelondro/util/MemoryStrategy.java, source/net/yacy/kelondro/util/ReverseMapIterator.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/kelondro/util/StandardMemoryStrategy.java, source/net/yacy/kelondro/workflow/AbstractThread.java, source/net/yacy/kelondro/workflow/InstantBlockingThread.java, source/net/yacy/kelondro/workflow/InstantBusyThread.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/dht/Dispatcher.java, source/net/yacy/peers/dht/FlatWordPartitionScheme.java, source/net/yacy/peers/dht/PeerSelection.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/peers/operation/yacySeedUploadFile.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/EventTracker.java, source/net/yacy/search/IndexingQueueEntry.java, source/net/yacy/search/Shutdown.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/RankingProfile.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java, source/net/yacy/search/solr/SolrServlet.java, source/net/yacy/upnp/Discovery.java, source/net/yacy/upnp/DiscoveryAdvertisement.java, source/net/yacy/upnp/DiscoveryListener.java, source/net/yacy/upnp/HttpResponse.java, source/net/yacy/upnp/devices/UPNPRootDevice.java, source/net/yacy/upnp/impls/InternetGatewayDevice.java, source/net/yacy/upnp/services/ISO8601Date.java, source/net/yacy/visualization/RasterPlotter.java, source/net/yacy/yacy.java, source/org/apache/tools/tar/TarInputStream.java |
Mon Aug 13 13:32:32 CEST 2012 by Michael Peter Christen | - renamed DoubleSolrConnector to MirrorSolrConnector and added a hit/miss/document cache to the MirrorSolrConnector. - more abstraction to SolrDocument in Connector interface - bugfixes in Solr field reader Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSelection.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/query/SnippetProcess.java |
Mon Aug 13 10:40:04 CEST 2012 by Michael Peter Christen | - added coordinate storage in solr schema - fixed shutdown process - fixed some solr-to-metadata reading - added a large number of metadata attributes in ViewFile.html Changed Files: defaults/solr/schema.xml, htroot/ViewFile.html, htroot/ViewFile.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/server/serverObjects.java, source/net/yacy/cora/document/ASCII.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/peers/dht/Dispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Fri Aug 10 13:26:51 CEST 2012 by Michael Peter Christen | - Implemented and integrated the URIMetadataNode object which is a metadata representation from the solr index. This shall replace metadata from the built-in database in the future. - added the Solr-driven metadata into the search index of YaCy which makes it now possible to run YaCy without the old metadata index. This is a major stept forward to a full migration to Solr. Changed Files: defaults/solr.keys.list, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, source/de/anomic/crawler/ResultURLs.java, source/net/yacy/kelondro/data/meta/URIMetadata.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/meta/URIReference.java, source/net/yacy/kelondro/data/meta/URIReferenceNode.java, source/net/yacy/kelondro/data/word/WordReference.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/index/DocumentReference.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/snippet/ResultEntry.java |
Fri Aug 10 07:45:43 CEST 2012 by Michael Peter Christen | better abstraction for result writers using controlled vocabularies and URIRefs Changed Files: source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/lod/Vocabulary.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/DCTerms.java, source/net/yacy/cora/lod/vocabulary/DublinCore.java, source/net/yacy/cora/lod/vocabulary/Foaf.java, source/net/yacy/cora/lod/vocabulary/Geo.java, source/net/yacy/cora/lod/vocabulary/HttpHeader.java, source/net/yacy/cora/lod/vocabulary/Owl.java, source/net/yacy/cora/lod/vocabulary/Rdf.java, source/net/yacy/cora/lod/vocabulary/YaCyMetadata.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/cora/services/federated/solr/SolrType.java, source/net/yacy/kelondro/data/meta/MetadataVocabulary.java |
Fri Aug 10 06:47:13 CEST 2012 by Michael Peter Christen | refactoring Changed Files: htroot/IndexFederated_p.java, htroot/api/schema_p.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/Schema.java, source/net/yacy/cora/services/federated/solr/ShardSelection.java, source/net/yacy/cora/services/federated/solr/SolrDoc.java, source/net/yacy/cora/storage/ConfigurationSet.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Thu Aug 09 18:06:48 CEST 2012 by Michael Peter Christen | added two response writer for embedded solr interface: a rss/opensearch writer and an enhanced solr xml writer. The enhanced solr writer has less configuration overhead than the original writer and should by slightly faster. The rss/opensearch writer is at this time slightly incomplete compared with the already existing rss search result form YaCy and also snippets are missing at this time. To test the new interface, open for example: http://localhost:8090/solr/select?wt=rss&q=olympia The wt-code for the new result writers are= wt=rss for opensearch wt=exml for the enhanced solr xml writer. Additionally, the SRU search parameters had been added to the solr interface which can now also be used for a normal solr/xml search. Changed Files: htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Sun Aug 05 15:49:27 CEST 2012 by orbiter | added more solr fields to integrate values from URIMetadataRow. All writings to the Metadata-DB are now also done to solr. This includes metadata transfer during search and rwi transfer. The new/added solr fields are: ## time when resource was loaded load_date_dt ## date until resource shall be considered as fresh fresh_date_dt ## id of the host, a 6-byte hash that is part of the document id host_id_s ## ids of referrer to this document referrer_id_ss ## the md5 of the raw source md5_s ## the name of the publisher of the document publisher_t ## the language used in the document; starts with primary language language_ss ## an external ranking value ranking_i ## the size of the raw source size_i ## number of links to audio resources audiolinkscount_i ## number of links to video resources videolinkscount_i ## number of links to application resources applinkscount_i Changed Files: defaults/solr.keys.list, htroot/IndexFederated_p.java, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, source/de/anomic/crawler/ResultURLs.java, source/de/anomic/crawler/retrieval/Response.java, source/net/yacy/cora/services/federated/solr/SolrDoc.java, source/net/yacy/cora/storage/ConfigurationSet.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/SolrField.java |
Mon Jul 30 10:38:23 CEST 2012 by Michael Peter Christen | snippet retrieval loading processes may use a smaller minimum load time value than crawling processes. This speeds up the search result preparation dramatically. Changed Files: htroot/Bookmarks.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/Load_RSS_p.java, htroot/ViewFile.java, htroot/ViewImage.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/webstructure.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/de/anomic/crawler/CrawlQueues.java, source/de/anomic/crawler/RSSLoader.java, source/de/anomic/data/ymark/YMarkAutoTagger.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java |
Fri Jul 27 12:13:53 CEST 2012 by Michael Peter Christen | Abstraction of HandleMap and HandleSet Changed Files: htroot/CrawlProfileEditor_p.java, htroot/Crawler_p.java, htroot/IndexControlRWIs_p.java, htroot/Load_RSS_p.java, htroot/Table_API_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.java, htroot/api/table_p.java, htroot/api/timeline.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/delete_ymark.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/import_ymark.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacysearch.java, source/de/anomic/crawler/Balancer.java, source/de/anomic/crawler/Cache.java, source/de/anomic/crawler/CrawlSwitchboard.java, source/de/anomic/crawler/NoticedURL.java, source/de/anomic/crawler/RSSLoader.java, source/de/anomic/crawler/RobotsTxt.java, source/de/anomic/crawler/ZURL.java, source/de/anomic/data/BlogBoard.java, source/de/anomic/data/BlogBoardComments.java, source/de/anomic/data/BookmarkDate.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/data/MessageBoard.java, source/de/anomic/data/UserDB.java, source/de/anomic/data/WorkTables.java, source/de/anomic/data/wiki/WikiBoard.java, source/de/anomic/data/ymark/YMarkAutoTagger.java, source/de/anomic/data/ymark/YMarkTables.java, source/net/yacy/cora/storage/HandleMap.java, source/net/yacy/cora/storage/HandleSet.java, source/net/yacy/cora/util/SpaceExceededException.java, source/net/yacy/dbtest.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BLOB.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Stack.java, source/net/yacy/kelondro/blob/Stacks.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/Cache.java, source/net/yacy/kelondro/index/Index.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/RAMIndex.java, source/net/yacy/kelondro/index/RAMIndexCluster.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/order/NaturalOrder.java, source/net/yacy/kelondro/rwi/AbstractIndex.java, source/net/yacy/kelondro/rwi/Index.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/rwi/IndexReader.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/rwi/TermSearch.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/dht/Dispatcher.java, source/net/yacy/peers/dht/FlatWordPartitionScheme.java, source/net/yacy/peers/dht/PeerSelection.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java |
Wed Jul 25 14:31:54 CEST 2012 by Michael Peter Christen | cleaned up classes and methods which are either superfluous at this time or will be superfluous or subject of complete redesign after the migration to solr. Removing these things now will make the transition to solr more simple. Changed Files: htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, htroot/env/templates/submenuBlacklist.template, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/yacy.java |
Wed Jul 25 01:53:47 CEST 2012 by Michael Peter Christen | Moved solr index-add method to the same method where the YaCy index is written. Also done some code-cleanup. Changed Files: htroot/IndexFederated_p.java, htroot/api/schema_p.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/IndexingQueueEntry.java, source/net/yacy/search/Shutdown.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/yacy.java |
Tue Jul 24 17:23:29 CEST 2012 by Michael Peter Christen | extended abstraction of local and remote solr index using one front-end for index administration and querying. Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/DoubleSolrConnector.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SnippetProcess.java |
Mon Jul 23 16:28:39 CEST 2012 by Michael Peter Christen | changed options in IndexFederated_p to switch on/off parts of the index individually. The settings are experimental and the values of the settings will be overwritten when an index migration from urldb to solr starts. Changed Files: defaults/yacy.init, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/yacysearchtrailer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/yacy.java |
Sun Jul 22 13:18:45 CEST 2012 by orbiter | - more abstraction for the RWI index as preparation for solr integration - added options in search index to switch parts of the index on or off Changed Files: htroot/Bookmarks.java, htroot/CrawlResults.java, htroot/IndexControlRWIs_p.html, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/api/yacydoc.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/data/URLAnalysis.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/kelondro/data/meta/URIMetadata.java, source/net/yacy/kelondro/data/meta/URIReference.java, source/net/yacy/kelondro/data/meta/URIReferenceNode.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java, source/net/yacy/yacy.java |
Thu Jul 19 11:34:05 CEST 2012 by Michael Peter Christen | added a solr search index - by default, a (empty) solr storage instance is created at SEGMENTS/solr_36 - the index is written if in /IndexFederated_p.html the flag "embedded solr search index" is switched on - a standard solr query interface is available now with a new servlet at http://127.0.0.1:8090/solr/select To test this, do the following: - switch to webportal mode - switch on the feature as described - do a crawl. this fills the solr index. The normal YaCy search will NOT work now! - do a solr query, like: http://127.0.0.1:8090/solr/select?q=*:* http://127.0.0.1:8090/solr/select?q=text_t:Help play with different search fields as you can see in /IndexFederated_p.html You can use the standard solr query attributes as described in http://wiki.apache.org/solr/SearchHandler Changed Files: htroot/solr/select.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/server/serverObjects.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java, source/net/yacy/search/solr/SolrServlet.java |
Fri Jul 13 21:15:38 CEST 2012 by Michael Peter Christen | collection of speed and memory saving hacks Changed Files: htroot/api/yacydoc.java, source/de/anomic/crawler/ResultImages.java, source/de/anomic/crawler/RobotsTxt.java, source/de/anomic/crawler/ZURL.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/logging/ConsoleOutHandler.java, source/net/yacy/kelondro/logging/Log.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/visualization/RasterPlotter.java |
Thu Jul 12 19:23:40 CEST 2012 by orbiter | reduced logging overhead (a bit) Changed Files: source/net/yacy/document/AbstractParser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/logging/Log.java |
Thu Jul 12 11:12:21 CEST 2012 by orbiter | replaced more size() > 0 by !isEmpty() Changed Files: htroot/CrawlStartScanner_p.java, htroot/IndexImportOAIPMH_p.java, htroot/Status.java, htroot/Table_API_p.java, htroot/yacysearchtrailer.java, source/de/anomic/crawler/Balancer.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/retrieval/Request.java, source/de/anomic/crawler/retrieval/Response.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/net/yacy/ai/example/SchwarzerPeter.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/search/index/SolrConfiguration.java |
Tue Jul 10 22:59:03 CEST 2012 by orbiter | - replaced all length() == 0 and size() == 0 with isEmpty() - replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be done automatically - implemented some isEmpty() methods Changed Files: htroot/BlacklistCleaner_p.java, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/ConfigAccounts_p.java, htroot/ConfigNetwork_p.java, htroot/CrawlResults.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Network.java, htroot/SettingsAck_p.java, htroot/Status.java, htroot/Surftips.java, htroot/Table_API_p.java, htroot/Tables_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/WatchWebStructure_p.java, htroot/WebStructurePicture_p.java, htroot/api/blacklists.java, htroot/api/blacklists_p.java, htroot/api/table_p.java, htroot/api/yacydoc.java, htroot/compare_yacy.java, htroot/yacy/message.java, htroot/yacy/search.java, htroot/yacysearchtrailer.java, libbuild/GitRevTask/GitRevTask.java, libbuild/svnRevNr/svnRevNrParser.java, source/de/anomic/crawler/CrawlProfile.java, source/de/anomic/crawler/CrawlSwitchboard.java, source/de/anomic/crawler/RobotsTxtEntry.java, source/de/anomic/crawler/RobotsTxtParser.java, source/de/anomic/crawler/retrieval/FTPLoader.java, source/de/anomic/crawler/retrieval/HTTPLoader.java, source/de/anomic/crawler/retrieval/Response.java, source/de/anomic/data/BlogBoard.java, source/de/anomic/data/BlogBoardComments.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/data/DidYouMean.java, source/de/anomic/data/MessageBoard.java, source/de/anomic/data/Translator.java, source/de/anomic/data/URLAnalysis.java, source/de/anomic/data/WorkTables.java, source/de/anomic/http/server/ChunkedInputStream.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/http/server/HTTPDemon.java, source/de/anomic/server/serverCore.java, source/de/anomic/server/serverObjects.java, source/de/anomic/tools/cryptbig.java, source/net/yacy/ai/example/Hanoi.java, source/net/yacy/cora/date/GenericFormatter.java, source/net/yacy/cora/date/ISO8601Formatter.java, source/net/yacy/cora/document/Classification.java, source/net/yacy/cora/document/JSONObject.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/language/phonetic/DoubleMetaphone.java, source/net/yacy/cora/language/phonetic/Metaphone.java, source/net/yacy/cora/language/phonetic/Soundex.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/services/federated/solr/SingleSolrConnector.java, source/net/yacy/cora/sorting/ClusteredScoreMap.java, source/net/yacy/cora/storage/KeyList.java, source/net/yacy/dbtest.java, source/net/yacy/document/Autotagging.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/StringBuilderComparator.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/WordCache.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/geolocation/GeonamesLocation.java, source/net/yacy/document/geolocation/Locations.java, source/net/yacy/document/geolocation/OpenGeoDBLocation.java, source/net/yacy/document/geolocation/OverarchingLocation.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ContentTransformer.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/Column.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/io/CharBuffer.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/kelondro/order/Base64Order.java, source/net/yacy/kelondro/rwi/AbstractReference.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/ByteBuffer.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/ReverseMapIterator.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacySeedUploadFile.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/EventTracker.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/RankingProfile.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/upnp/Discovery.java, source/net/yacy/upnp/DiscoveryAdvertisement.java, source/net/yacy/upnp/DiscoveryListener.java, source/net/yacy/upnp/HttpResponse.java, source/net/yacy/upnp/devices/UPNPRootDevice.java, source/net/yacy/yacy.java |
Mon Jul 09 18:58:33 CEST 2012 by sixcooler | bump to httpclient-4.2.1 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/dependencies.txt, lib/httpclient-4.2.1.License, lib/httpclient-4.2.1.jar, lib/httpcore-4.2.1.License, lib/httpcore-4.2.1.jar, lib/httpmime-4.2.1.License, lib/httpmime-4.2.1.jar, nbproject/project.xml |
Mon Jul 09 11:14:50 CEST 2012 by orbiter | refactoring of query attribute variable names for better consistency with (next) stored query words Changed Files: htroot/AccessTracker_p.java, htroot/yacy/search.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SnippetProcess.java |
Commit | Description |
---|---|
Fri Aug 24 21:44:22 CEST 2012 by orbiter | small fixes Changed Files: htroot/ConfigBasic.java, htroot/IndexControlRWIs_p.java, htroot/IndexCreateQueues_p.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/document/parser/swfParser.java |
Fri Aug 24 14:15:40 CEST 2012 by orbiter | fixed concurrent query Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java |
Fri Aug 24 14:13:42 CEST 2012 by orbiter | fixed generation of ontologies from index enumerations Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java, source/net/yacy/search/index/Segment.java |
Wed Aug 22 01:55:39 CEST 2012 by Michael Peter Christen | fix for NPE during host navigation computation Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java |
Wed Aug 22 00:48:37 CEST 2012 by Michael Peter Christen | fixed GSA format Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java |
Tue Aug 21 02:39:28 CEST 2012 by Michael Peter Christen | gsa bugfix for date parser Changed Files: htroot/gsa/searchresult.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java |
Tue Aug 21 01:57:46 CEST 2012 by Michael Peter Christen | fixes for gsa result format Changed Files: htroot/gsa/searchresult.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java |
Mon Aug 20 14:16:37 CEST 2012 by Michael Peter Christen | fixed double-check Changed Files: source/net/yacy/search/query/RWIProcess.java |
Mon Aug 20 12:21:36 CEST 2012 by Michael Peter Christen | bugfix for remote search when search is done to solr Changed Files: source/net/yacy/peers/Protocol.java |
Sun Aug 19 08:43:56 CEST 2012 by Michael Peter Christen | fix for http://bugs.yacy.net/view.php?id=206 Changed Files: source/net/yacy/search/query/RWIProcess.java |
Sat Aug 18 13:05:27 CEST 2012 by orbiter | fixes for putDocument and putMetadata Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java |
Thu Aug 16 07:47:52 CEST 2012 by orbiter | fix for Index out of bounds exception in Network servlet Changed Files: htroot/Network.java |
Wed Aug 15 14:23:21 CEST 2012 by Lotus | fix xss bug #204 Changed Files: htroot/Network.java |
Tue Jul 31 23:49:07 CEST 2012 by Michael Peter Christen | bad hack to prevent a bug appearing in solr Changed Files: defaults/solr/solrconfig.xml, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Mon Jul 30 14:51:01 CEST 2012 by Michael Peter Christen | fix for a NPE Changed Files: source/net/yacy/repository/Blacklist.java |
Tue Jul 24 17:29:32 CEST 2012 by Michael Peter Christen | bugfix for a NPE Changed Files: source/net/yacy/peers/Protocol.java |
Mon Jul 23 23:40:50 CEST 2012 by Michael Peter Christen | fixed node type calculation for principal peers Changed Files: source/net/yacy/peers/Protocol.java |
Mon Jul 23 00:36:18 CEST 2012 by Michael Peter Christen | fix for http://bugs.yacy.net/view.php?id=202 Changed Files: htroot/ViewFile.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java |
Mon Jul 09 14:32:35 CEST 2012 by orbiter | fix for RSS reader Changed Files: source/net/yacy/cora/document/RSSReader.java |
Commit | Description |
---|---|
Fri Aug 24 23:59:10 CEST 2012 by orbiter | release 1.1 Changed Files: build.properties |
Fri Aug 24 23:04:19 CEST 2012 by orbiter | added a direct access to solr search api to enhance the visibility if the embedded solr Changed Files: htroot/env/templates/header.template |
Fri Aug 24 18:24:24 CEST 2012 by orbiter | Merge commit 'c2341a175fdd755a34965ff63c7ea437b380352d' Changed Files: source/net/yacy/cora/protocol/ftp/FTPClient.java |
Fri Aug 24 17:45:14 CEST 2012 by David Rubio | Fixed a bug that prevented Yacy from indexing files with non ASCII filenames in FTP servers. Previously Yacy could read file listings in UTF-8, but couldn't send commands to the FTP server in UTF-8 (the second byte of every multi-byte character was ignored), which caused a lot of errors on the server side. Now it handles UTF-8 correctly. Changed Files: source/net/yacy/cora/protocol/ftp/FTPClient.java |
Fri Aug 24 12:18:30 CEST 2012 by orbiter | omit xml parsing when using the embedded solr server Changed Files: source/net/yacy/cora/services/federated/solr/EmbeddedSolrConnector.java, source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java |
Thu Aug 23 11:53:54 CEST 2012 by orbiter | added the QueryResponse query(SolrParams params) method to the SolrServerConnector which is necessary to use facets in solr search. Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java |
Thu Aug 23 09:51:45 CEST 2012 by orbiter | redesign of YaCySchema and SolrDoc handling Changed Files: source/de/anomic/crawler/ZURL.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java |
Thu Aug 23 09:30:11 CEST 2012 by orbiter | refctoring Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EmbeddedSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServlet.java, source/net/yacy/search/index/Fulltext.java |
Wed Aug 22 23:50:40 CEST 2012 by Michael Peter Christen | log queries anonymous from gsa+solr requests Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/query/AccessTracker.java |
Wed Aug 22 17:37:34 CEST 2012 by Michael Peter Christen | added snippet computation to solr/rss and gsa result writer Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/search/query/SnippetProcess.java |
Wed Aug 22 16:30:33 CEST 2012 by Michael Peter Christen | - reduced memory usage in index transmission using a transformation of Node to Row objects - removed peerDeparture in solr remote search in case that peer does not answer (this may be normal because it is allowed to switch this off) Changed Files: htroot/IndexControlRWIs_p.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/search/index/Fulltext.java |
Wed Aug 22 01:56:13 CEST 2012 by Michael Peter Christen | re-activated audio and video search because they obviously work (!) Changed Files: defaults/yacy.init |
Wed Aug 22 00:48:03 CEST 2012 by Michael Peter Christen | corrected solr query syntax Changed Files: source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/query/QueryParams.java |
Wed Aug 22 00:31:14 CEST 2012 by Michael Peter Christen | - enhanced caching after search queries to solr - reduced caching after short memory Changed Files: source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java |
Tue Aug 21 23:52:56 CEST 2012 by orbiter | sorted the solr schema into mandatory and optional fields; reduced number of used field to reduce solr index size Changed Files: defaults/solr.keys.list, source/net/yacy/search/index/YaCySchema.java |
Tue Aug 21 21:03:26 CEST 2012 by orbiter | fix from gaston in http://forum.yacy-websuche.de/viewtopic.php?p=26909#p26909 Changed Files: source/net/yacy/search/index/SolrConfiguration.java |
Tue Aug 21 18:18:30 CEST 2012 by orbiter | removed unused classes Changed Files: source/net/yacy/kelondro/data/meta/URIMetadata.java |
Mon Aug 20 17:10:48 CEST 2012 by Michael Peter Christen | added authorization-based maximum results limitation to solr and gsa search Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java |
Mon Aug 20 16:53:21 CEST 2012 by Michael Peter Christen | added gzip encoding to solr2sor http interface, client side (server already works) Changed Files: source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java |
Mon Aug 20 13:02:29 CEST 2012 by Michael Peter Christen | added a tooltip for search navigation to mention that search pages can be navigated using the TAB key Changed Files: htroot/js/yacysearch.js |
Mon Aug 20 12:50:51 CEST 2012 by Michael Peter Christen | gsa format update Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/query/QueryParams.java |
Mon Aug 20 01:29:15 CEST 2012 by Michael Peter Christen | more abstraction and less parameter overhead for remote search Changed Files: htroot/yacy/search.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/search/query/SearchEvent.java |
Sun Aug 19 13:17:03 CEST 2012 by Michael Peter Christen | code simplifications Changed Files: htroot/Connections_p.java, source/de/anomic/server/serverSwitch.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Sun Aug 19 08:44:39 CEST 2012 by Michael Peter Christen | removed strange assert statements and simplified code in metadata transformation Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java |
Sat Aug 18 23:59:25 CEST 2012 by orbiter | refactoring in remote search and stub for remote node peer selection Changed Files: source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/dht/PeerSelection.java, source/net/yacy/search/query/SearchEvent.java |
Sat Aug 18 19:36:21 CEST 2012 by orbiter | - get nice text_t values from metadata conversions that are stored into solr as fulltext search index. - added slow migration from old metadata to solr index entries: each entry from the old metadata is removed from that data structure and written into solr. Changed Files: source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/SolrConfiguration.java |
Sat Aug 18 17:48:20 CEST 2012 by orbiter | reduced sleep times Changed Files: source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java |
Sat Aug 18 15:39:14 CEST 2012 by orbiter | added ramaining iteration methods for solr in fulltext class Changed Files: htroot/IndexControlURLs_p.java, source/net/yacy/cora/order/CloneableIterator.java, source/net/yacy/search/index/Fulltext.java |
Sat Aug 18 14:22:00 CEST 2012 by orbiter | hack to removed StringBuilder overhead in query construction Changed Files: source/net/yacy/cora/services/federated/solr/SolrServerConnector.java |
Sat Aug 18 13:45:37 CEST 2012 by orbiter | reduced solr cache sizes to check if that solves memory problems a bit Changed Files: defaults/solr/solrconfig.xml |
Sat Aug 18 13:18:51 CEST 2012 by orbiter | explicit double-check in transferURL Changed Files: htroot/yacy/transferURL.java |
Sat Aug 18 10:28:40 CEST 2012 by orbiter | reverted bf55f6917652909f8eb465ccefd1f7ccb4c4d364 to have a fall-back option in case that memory problems as reported in http://forum.yacy-websuche.de/viewtopic.php?p=26901#p26901 for full-solr installation are too strong and we have to work with an 'small memory footprint' peer system. Changed Files: source/net/yacy/search/index/Fulltext.java |
Fri Aug 17 18:22:56 CEST 2012 by Michael Peter Christen | added concurrent iterator methods to the solr connectors Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Fri Aug 17 17:28:27 CEST 2012 by Michael Peter Christen | refactoring Changed Files: htroot/IndexFederated_p.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java |
Fri Aug 17 17:17:00 CEST 2012 by Michael Peter Christen | better check for bad urls in url transmission Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataRow.java |
Fri Aug 17 17:05:46 CEST 2012 by Michael Peter Christen | added deleteByQuery to solr connectors Changed Files: .classpath, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java |
Fri Aug 17 15:46:26 CEST 2012 by Michael Peter Christen | removed write methods to old metadata file type; all metadata now goes to solr Changed Files: source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java |
Fri Aug 17 15:33:02 CEST 2012 by Michael Peter Christen | refactoring Changed Files: htroot/IndexControlURLs_p.java, htroot/YBRFetch_p.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java |
Fri Aug 17 15:11:21 CEST 2012 by Michael Peter Christen | upgrade to solr 3.6.1 Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/apache-solr-core-3.6.1.License, lib/apache-solr-core-3.6.1.jar, lib/apache-solr-solrj-3.6.1.License, lib/apache-solr-solrj-3.6.1.jar |
Fri Aug 17 14:34:31 CEST 2012 by Michael Peter Christen | bugfix for solr connector, possibly a cause for http://forum.yacy-websuche.de/viewtopic.php?p=26893#p26893 Changed Files: source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java |
Fri Aug 17 14:22:07 CEST 2012 by Michael Peter Christen | enhanced snippet fetch - removed a bug that caused documents to be parsed even if a solr text was available Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/TextSnippet.java |
Fri Aug 17 11:47:39 CEST 2012 by cominch | local robots.txt: disallow external crawlers to follow the URL proxy Changed Files: htroot/robots.txt |
Fri Aug 17 01:21:56 CEST 2012 by Michael Peter Christen | using the solr search index to concurrently search within solr and the rwis during local search requests. Changed Files: source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java |
Thu Aug 16 23:05:37 CEST 2012 by Michael Peter Christen | added clear-text search words in query params Changed Files: htroot/yacy/search.java, htroot/yacysearch.java, source/net/yacy/search/query/QueryParams.java |
Thu Aug 16 22:35:19 CEST 2012 by Michael Peter Christen | - added a content-encoding: gzip to streamed http server responses - finish and close streamed http responses immediately - this applies only to the solr interface which should be much faster now! Changed Files: source/de/anomic/http/server/HTTPDFileHandler.java |
Thu Aug 16 17:49:35 CEST 2012 by Michael Peter Christen | doctype2mime fix, influences metadata conversion between old metadata and solr Changed Files: defaults/httpd.mime, source/de/anomic/crawler/retrieval/Response.java, source/net/yacy/cora/document/Classification.java |
Thu Aug 16 17:38:10 CEST 2012 by cominch | changed local robots.txt to prevent external crawlers to submit random search queries Changed Files: htroot/robots.txt |
Thu Aug 16 17:24:25 CEST 2012 by Michael Peter Christen | more attempts to clean the index (cleaning is faster then) Changed Files: source/net/yacy/peers/Network.java |
Thu Aug 16 17:11:54 CEST 2012 by Michael Peter Christen | fixed some peer-ping connection details - larger time-out - removed too old seedlist - fixed a bug in connection test Changed Files: defaults/yacy.network.freeworld.unit, htroot/ViewProfile.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/Protocol.java |
Thu Aug 16 07:28:38 CEST 2012 by orbiter | addon to e74d66e28cce7b9674ad5011e5db7970ccaf5635 (removed htmlparser.jar): for Mac App Changed Files: addon/YaCy.app/Contents/Info.plist |
Tue Aug 14 13:29:11 CEST 2012 by Michael Peter Christen | replaced yacy xml encoding by solr xml encoding Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java |
Tue Aug 14 13:19:29 CEST 2012 by Michael Peter Christen | enhanced GSA and RSS output format: corrected date, added some missing fields, added xml encoding for utf8 Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java |
Tue Aug 14 12:40:26 CEST 2012 by Michael Peter Christen | added a very rudimentary, incomplete, non-verified GSA response writer for solr. Try this: http://localhost:8090/gsa/searchresult?q=pdf&site=col1&num=10 Changed Files: .classpath, htroot/gsa/searchresult.java, htroot/solr/select.java, source/de/anomic/server/serverObjects.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Tue Aug 14 11:12:50 CEST 2012 by Michael Peter Christen | - added xslt support for solr result formats. try i.e. http://localhost:8090/solr/select?q=*:*&start=0&rows=10&wt=xslt&tr=json.xsl - added servlet-side mime-type configuration for streamed servlets. this is used for the result formatters in solr result formats Changed Files: defaults/solr/xslt/example.xsl, defaults/solr/xslt/json.xsl, htroot/solr/select.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/server/serverCore.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Tue Aug 14 10:09:46 CEST 2012 by cominch | augmented browsing: remove htmlparser library Changed Files: .classpath, build.xml |
Tue Aug 14 10:06:12 CEST 2012 by cominch | augmented browsing: replace htmlparser by jsoup, which is more stable and reliable Changed Files: .classpath, build.xml, defaults/yacy.init, lib/jsoup-1.6.3.jar, source/net/yacy/interaction/AugmentHtmlStream.java |
Mon Aug 13 16:57:53 CEST 2012 by cominch | added a possibility to define a custom network definition URL for remote management Changed Files: htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java |
Mon Aug 13 14:01:45 CEST 2012 by Michael Peter Christen | ups Changed Files: source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java |
Mon Aug 13 11:13:53 CEST 2012 by Michael Peter Christen | another fix to the Solr metadata reading process and to the shutdown process Changed Files: source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java |
Sat Aug 11 01:05:40 CEST 2012 by Michael Peter Christen | removed unused class Changed Files: source/net/yacy/search/index/MetadataRepository.java |
Fri Aug 10 18:10:25 CEST 2012 by orbiter | tried to prevent calls to bad-hack getSize() method and reduced overhead of that method a bit. Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java |
Fri Aug 10 15:44:37 CEST 2012 by orbiter | patch from hint in http://forum.yacy-websuche.de/viewtopic.php?p=26858#p26858 from gaston Changed Files: htroot/yacy/transferRWI.java |
Fri Aug 10 15:39:10 CEST 2012 by orbiter | changed behaviour of metadata storage: in case that any solr is attached, the metadata is not written to the metadata-db, even if it is enabled but instead to solr. This prevents that metadata is written in two store systems at the same time. It is also the next step to migrate the current metadata-db to solr. Changed Files: htroot/IndexControlRWIs_p.java, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java |
Fri Aug 10 14:47:44 CEST 2012 by orbiter | removed unused classes Changed Files: |
Fri Aug 10 09:48:15 CEST 2012 by Michael Peter Christen | more abstraction of the YaCySchema -> Opensearch matching process Changed Files: htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java |
Fri Aug 10 07:58:45 CEST 2012 by Michael Peter Christen | more abstraction for solr query params parsing Changed Files: htroot/solr/select.java |
Fri Aug 10 07:51:57 CEST 2012 by Michael Peter Christen | set the title every time, it is possible that it has changed Changed Files: htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java |
Wed Aug 08 09:35:45 CEST 2012 by ????????? ?????? ????????? | Fix an error in Russian translation: "can not" => "can". Changed Files: locales/ru.lng |
Mon Aug 06 17:58:09 CEST 2012 by Michael Peter Christen | replaced the multivalue generic string field name suffix _ss by _txt because _ss is not part of the standard solr example schema. Changed Files: defaults/solr.keys.list, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/SolrField.java |
Wed Aug 01 00:14:56 CEST 2012 by Michael Peter Christen | added the JSON response writer to solr interface, add &wt=json to the servlet GET properties to use this format Changed Files: htroot/solr/select.java |
Tue Jul 31 23:23:16 CEST 2012 by sixcooler | prevent merge of blobs that can't be handled in memory Changed Files: source/net/yacy/kelondro/blob/ArrayStack.java |
Mon Jul 30 12:39:47 CEST 2012 by Michael Peter Christen | nowrap from gaston in forum http://forum.yacy-websuche.de/viewtopic.php?p=26815#p26815 Changed Files: htroot/Network.html |
Fri Jul 27 04:11:52 CEST 2012 by sixcooler | check content domain fix: search image/media should not show pages containing image/media search text should show all/text but image/media Changed Files: source/net/yacy/search/query/RWIProcess.java |
Thu Jul 26 18:09:40 CEST 2012 by sixcooler | close augmented stream if filled from cache to get its content use augmented stream if proxyAugmentation is set only Changed Files: source/de/anomic/http/server/HTTPDProxyHandler.java |
Thu Jul 26 10:05:06 CEST 2012 by Michael Peter Christen | better calculation of possible saving in HeapReader index data structure Changed Files: source/net/yacy/kelondro/blob/HeapReader.java |
Wed Jul 25 21:34:23 CEST 2012 by Michael Peter Christen | documentation/comments Changed Files: source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/index/HandleMap.java |
Wed Jul 25 15:35:13 CEST 2012 by sixcooler | no translation of queue-links Changed Files: locales/cn.lng, locales/de.lng, locales/ru.lng, locales/uk.lng |
Tue Jul 24 22:16:56 CEST 2012 by Michael Peter Christen | cleanup Changed Files: htroot/yacysearch.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/search/Switchboard.java |
Mon Jul 23 21:43:14 CEST 2012 by Michael Peter Christen | added user-authentication protection to solr search (same as implemented for yacysearch) Changed Files: htroot/solr/select.java |
Mon Jul 23 21:31:12 CEST 2012 by Michael Peter Christen | better explain how to access the embedded solr Changed Files: htroot/IndexControlRWIs_p.html, htroot/IndexFederated_p.html |
Mon Jul 23 00:00:40 CEST 2012 by reger | removed localized number formatting from num-results_totalcount response (this is only used in xml and json where localized format is not valid) Changed Files: htroot/yacysearch.java, htroot/yacysearchtrailer.java |
Fri Jul 20 11:47:50 CEST 2012 by orbiter | patches to ensure that solr connectors are active ony if they have a solr object assigned and vice versa Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, source/net/yacy/search/Switchboard.java |
Fri Jul 20 11:40:33 CEST 2012 by orbiter | embedded solr is only initiated if it is activated with IndexFederated_p.html Changed Files: htroot/IndexFederated_p.java, htroot/solr/select.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/yacy.java |
Fri Jul 20 09:04:02 CEST 2012 by Michael Peter Christen | source change in classpath Changed Files: .classpath |
Fri Jul 20 08:53:12 CEST 2012 by Lotus | partial html fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4454 Changed Files: htroot/yacysearchitem.html |
Fri Jul 20 00:59:58 CEST 2012 by orbiter | added classpath for htroot/solr Changed Files: .classpath |
Sat Jul 14 16:28:14 CEST 2012 by Michael Peter Christen | allow larger log entries Changed Files: source/net/yacy/kelondro/logging/Log.java |
Sat Jul 14 13:11:04 CEST 2012 by Michael Peter Christen | removed a crawler overhead (terminated loop which searches greatest stack that has zero-waiting urls). This should cause a slightly faster crawl for crawl stacks with many different domains in the crawl queue. Changed Files: source/de/anomic/crawler/Balancer.java |
Sat Jul 14 13:09:44 CEST 2012 by Michael Peter Christen | enhancement in internal data organization which should generate less synchronizations in database access Changed Files: source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/Records.java |
Thu Jul 12 19:54:54 CEST 2012 by orbiter | less usage of generic logger to avoid logger generation overhead Changed Files: htroot/yacysearch.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/search/query/SnippetProcess.java |
Thu Jul 12 19:42:42 CEST 2012 by orbiter | prevent enqueueing of non-loggeable logging entries Changed Files: source/net/yacy/kelondro/logging/Log.java |
Thu Jul 12 02:08:11 CEST 2012 by Michael Peter Christen | reduction of logging to prevent too much IO caused be logging Changed Files: source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/ZURL.java, source/net/yacy/search/Switchboard.java |
Thu Jul 12 01:23:04 CEST 2012 by Michael Peter Christen | fixed a memory leak inside the logger which appeared if the log was writter faster that the logger is able to print this out to its out stream. A very large collection of unwritten log outputs had been seen during strong crawling. The new ArrayBlockingQueue is limited to prevent this case. Changed Files: source/net/yacy/kelondro/logging/Log.java |
Wed Jul 11 23:18:57 CEST 2012 by Michael Peter Christen | added creation of subpath pattern when crawl start is 'from file' Changed Files: htroot/Crawler_p.java |
Tue Jul 10 17:39:56 CEST 2012 by orbiter | fix for url matcher of multiple amp& in an url, see: http://forum.yacy-websuche.de/viewtopic.php?f=8&t=4439&p=26650#p26650 Changed Files: source/net/yacy/cora/document/MultiProtocolURI.java |
Tue Jul 10 13:08:16 CEST 2012 by Roland 'Quix0r' Haeder | - removed cleaning of blacklist cache on startup - added cleaning of blacklist cache if cache is modified in interface - extended cache saving to all cache types - moved cache location to DATA/LISTS - fixed static file path which was relative to the application path but should be relative to data path - which is different in debian and mac implementations Changed Files: bin/localsearch.sh, htroot/Blacklist_p.java, source/de/anomic/data/ListManager.java, source/net/yacy/repository/Blacklist.java |
Tue Jul 10 12:01:20 CEST 2012 by orbiter | using SwitchboardConstants for solr attributes Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/QueryParams.java |