org.eclipse.smila.connectivity.framework.crawler.web.http
Class RobotRulesParser.RobotRuleSet

java.lang.Object
  org.eclipse.smila.connectivity.framework.crawler.web.http.RobotRulesParser.RobotRuleSet

Enclosing class:

RobotRulesParser

public static class RobotRulesParser.RobotRuleSet
extends java.lang.Object

This class holds the rules which were parsed from a robots.txt file, and can test paths against those rules.

Constructor Summary

RobotRulesParser.RobotRuleSet()

Method Summary

long	getCrawlDelay() Get Crawl-Delay, in milliseconds.
long	getExpireTime() Get expire time.
boolean	isAllowed(java.lang.String path) Returns false if the robots.txt file prohibits us from accessing the given path, or true otherwise.
void	setCrawlDelay(long crawlDelay) Set Crawl-Delay, in milliseconds.
void	setExpireTime(long expireTime) Change when the rule set goes stale.
java.lang.String	toString() Returns readable representation of robot rules.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RobotRulesParser.RobotRuleSet

public RobotRulesParser.RobotRuleSet()

Method Detail

setExpireTime

public void setExpireTime(long expireTime)

Change when the rule set goes stale.

Parameters:

expireTime - time when the rule set goes stale

getExpireTime

public long getExpireTime()

Get expire time.

Returns:

long

getCrawlDelay

public long getCrawlDelay()

Get Crawl-Delay, in milliseconds. This returns -1 if not set.

Returns:

long crawlDelay

setCrawlDelay

public void setCrawlDelay(long crawlDelay)

Set Crawl-Delay, in milliseconds.

Parameters:

crawlDelay - long

isAllowed

public boolean isAllowed(java.lang.String path)

Returns false if the robots.txt file prohibits us from accessing the given path, or true otherwise.

Parameters:

path - String

Returns:

boolean

toString

public java.lang.String toString()

Returns readable representation of robot rules.

Overrides:

toString in class java.lang.Object

Returns:

String