org.eclipse.smila.connectivity.framework.crawler.web.fetcher
Class Fetcher

java.lang.Object
  org.eclipse.smila.connectivity.framework.crawler.web.configuration.Configured
      org.eclipse.smila.connectivity.framework.crawler.web.fetcher.Fetcher

All Implemented Interfaces:

Configurable

public class Fetcher
extends Configured

Class that fetches the link and updates crawler status.

Field Summary

Fields inherited from class org.eclipse.smila.connectivity.framework.crawler.web.configuration.Configured

_configuration

Constructor Summary

Fetcher(Configuration configuration, ParserManager parserManager, CrawlerPerformanceCounterHelper<WebCrawlerPerformanceAgent> performanceCounters)
Creates object with given Configuration.

Method Summary

FetcherOutput	fetch(Outlink link, FilterProcessor filterProcessor, java.util.Set<Outlink> linksDone) Fetches the page and parses the link.
long	getBytes() Returns amount of bytes fetched so far.
int	getErrors() Returns number of fetching errors happened so far.
int	getPages() Returns number of fetched pages so far.
void	setBytes(long bytes) Assigns the amount of bytes fetched.
void	setErrors(int errors) Assigns number of fetching errors.
void	setPages(int pages) Assigns number of fetched pages.

Methods inherited from class org.eclipse.smila.connectivity.framework.crawler.web.configuration.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Fetcher

public Fetcher(Configuration configuration,
               ParserManager parserManager,
               CrawlerPerformanceCounterHelper<WebCrawlerPerformanceAgent> performanceCounters)

Creates object with given Configuration.

Parameters:

configuration - Configuration

performanceCounters - the performance counters

parserManager - parser manager

Method Detail

fetch

public FetcherOutput fetch(Outlink link,
                           FilterProcessor filterProcessor,
                           java.util.Set<Outlink> linksDone)

Fetches the page and parses the link.

Parameters:

link - link to fetch

filterProcessor - filters to perform

linksDone - list of already crawled links

Returns:

FetcherOutput

getBytes

public long getBytes()

Returns amount of bytes fetched so far.

Returns:

long

setBytes

public void setBytes(long bytes)

Assigns the amount of bytes fetched.

Parameters:

bytes - long

getErrors

public int getErrors()

Returns number of fetching errors happened so far.

Returns:

errors

setErrors

public void setErrors(int errors)

Assigns number of fetching errors.

Parameters:

errors - errors

getPages

public int getPages()

Returns number of fetched pages so far.

Returns:

pages

setPages

public void setPages(int pages)

Assigns number of fetched pages.

Parameters:

pages - pages