org.eclipse.smila.importing.crawler.web.filter

Class FilterConfiguration

java.lang.Object

org.eclipse.smila.importing.crawler.web.filter.FilterConfiguration

public class FilterConfiguration
extends java.lang.Object

web crawler filter configuration, used by @ LinkFilter} class.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FilterConfiguration.StayOn Possible values for 'stayOn' parameter.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	EXCLUDE_PATTERNS
static java.lang.String	FOLLOW_REDIRECTS
static java.lang.String	INCLUDE_PATTERNS
static java.lang.String	MAX_CRAWL_DEPTH
static java.lang.String	MAX_REDIRECTS
static java.lang.String	STAY_ON Name of the task parameter that signals whether to stay on host resp. domain.
static java.lang.String	URL_PATTERNS

Constructor Summary

Constructors

Constructor and Description
FilterConfiguration(AnyMap filterConfig)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	followRedirects()
long	getMaxCrawlDepth()
long	getMaxRedirects()
RegexPatternMatcher	getUrlPatternMatcher()
boolean	isStayOnDomain()
boolean	isStayOnHost()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_CRAWL_DEPTH

public static final java.lang.String MAX_CRAWL_DEPTH

See Also:

Constant Field Values

FOLLOW_REDIRECTS

public static final java.lang.String FOLLOW_REDIRECTS

See Also:

Constant Field Values

MAX_REDIRECTS

public static final java.lang.String MAX_REDIRECTS

See Also:

Constant Field Values

STAY_ON

public static final java.lang.String STAY_ON

Name of the task parameter that signals whether to stay on host resp. domain.

See Also:

Constant Field Values

URL_PATTERNS

public static final java.lang.String URL_PATTERNS

See Also:

Constant Field Values

INCLUDE_PATTERNS

public static final java.lang.String INCLUDE_PATTERNS

See Also:

Constant Field Values

EXCLUDE_PATTERNS

public static final java.lang.String EXCLUDE_PATTERNS

See Also:

Constant Field Values

Constructor Detail

FilterConfiguration

public FilterConfiguration(AnyMap filterConfig)

Parameters:

filterConfig - filter section from file crawler configuration.

Method Detail

getUrlPatternMatcher

public RegexPatternMatcher getUrlPatternMatcher()

Returns:

matcher for checking include and exclude patterns of URLs.

followRedirects

public boolean followRedirects()

Returns:

'true' if we should follow redirects, 'false' otherwise.

getMaxRedirects

public long getMaxRedirects()

Returns:

the maximum number of allowed redirects when following links.

getMaxCrawlDepth

public long getMaxCrawlDepth()

Returns:

the maximum depth when following links.

isStayOnHost

public boolean isStayOnHost()

Returns:

whether URLs that would leave the host should be crawled.

isStayOnDomain

public boolean isStayOnDomain()

Returns:

whether URLs that would leave the domain should be crawled.