org.eclipse.smila.importing.crawler.web.utils

Class WebCrawlerConfiguration

java.lang.Object

org.eclipse.smila.importing.crawler.web.utils.WebCrawlerConfiguration

public class WebCrawlerConfiguration
extends java.lang.Object

Configuration for the web crawler.

Constructor Summary

Constructors

Constructor and Description
WebCrawlerConfiguration() reads the configuration.

Method Summary

Methods

Modifier and Type	Method and Description
HttpHost	getProxyHost()
java.lang.Integer	getSocketTimeout()
java.lang.String	getUserAgent()
boolean	isAllowAllOn40xForRobots()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WebCrawlerConfiguration

public WebCrawlerConfiguration()

reads the configuration.

Method Detail

getProxyHost

public HttpHost getProxyHost()

Returns:

the configured Proxy Host or null if no proxy is configured.

getSocketTimeout

public java.lang.Integer getSocketTimeout()

Returns:

the configured socket timeout (in ms), -1 if no socket timeout has been configured.

getUserAgent

public java.lang.String getUserAgent()

Returns:

configured userAgent string if any, else WebCrawlerConstants.DEFAULT_USERAGENT.

isAllowAllOn40xForRobots

public boolean isAllowAllOn40xForRobots()