net.yacy.crawler.robots
Classes
RobotsTxt
RobotsTxt.CheckEntry
RobotsTxt.DomSync
RobotsTxtEntry
RobotsTxtParser