org.eclipse.smila.importing.crawler.web.utils

Class RobotsTxt

java.lang.Object

org.eclipse.smila.importing.crawler.web.utils.RobotsTxt

public class RobotsTxt
extends java.lang.Object

parses a robots.txt file and checks if paths are allowed. See [1] http://www.robotstxt.org/norobots-rfc.txt for syntax definition

Constructor Summary

Constructors

Constructor and Description
RobotsTxt() create empty instance allowing everything.
RobotsTxt(boolean disallowAll) create empty instance allowing or forbidding everything.
RobotsTxt(byte[] serialized) create instance from serialized version, as created by asBinary().
RobotsTxt(java.lang.String... disallows) create instance from given paths, used for tests.
RobotsTxt(java.lang.String userAgent, java.io.InputStream stream) create instance by reading "Disallow" entries from stream that apply to the given user-agent.

Method Summary

Methods

Modifier and Type	Method and Description
byte[]	asBinary() serialized to byte array for cluster serialization.
boolean	isAllowed(java.lang.String path) check if the path is allowed for the user agent in the parsed robots.txt file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RobotsTxt

public RobotsTxt()

create empty instance allowing everything.

RobotsTxt

public RobotsTxt(boolean disallowAll)

create empty instance allowing or forbidding everything.

RobotsTxt

public RobotsTxt(java.lang.String userAgent,
         java.io.InputStream stream)
          throws java.io.IOException

create instance by reading "Disallow" entries from stream that apply to the given user-agent.

Throws:

java.io.IOException - error reading the stream

RobotsTxt

public RobotsTxt(byte[] serialized)

create instance from serialized version, as created by asBinary().

RobotsTxt

public RobotsTxt(java.lang.String... disallows)

create instance from given paths, used for tests.

Method Detail

isAllowed

public boolean isAllowed(java.lang.String path)

check if the path is allowed for the user agent in the parsed robots.txt file.

Parameters:

path - path portion or the URL to check.

Returns:

true if the user agent is allowed to see this path.

asBinary

public byte[] asBinary()

serialized to byte array for cluster serialization.