org.eclipse.smila.importing.crawler.web

Interface LinkFilter

All Known Implementing Classes:

DefaultLinkFilter

public interface LinkFilter

interface for LinkFilter services. The LinkFilter is called on the result of the LinkExtractor to select only those links that should really be followed in follow-up tasks.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allowRedirectLink(java.lang.String link, java.lang.String originalUrl, WebCrawlingContext context) Check if it is allowed to follow a given redirect link.
java.util.Collection<Record>	filterExtractedLinks(java.util.Collection<Record> extractedLinks, java.lang.String originalUrl, WebCrawlingContext context) filter links extracted from given source URL.

Method Detail

filterExtractedLinks

java.util.Collection<Record> filterExtractedLinks(java.util.Collection<Record> extractedLinks,
                                                java.lang.String originalUrl,
                                                WebCrawlingContext context)
                                                  throws WebCrawlerException

filter links extracted from given source URL.

Parameters:

extractedLinks - result from LinkExtractor service.

originalUrl - the source URL from which the links were extracted

context - the WebCrawlingContext.

Returns:

links to follow in follow-up tasks

Throws:

WebCrawlerException

allowRedirectLink

boolean allowRedirectLink(java.lang.String link,
                        java.lang.String originalUrl,
                        WebCrawlingContext context)
                          throws WebCrawlerException

Check if it is allowed to follow a given redirect link.

Parameters:

link - a String containing the link to be checked

originalUrl - the original URL that was redirected.

context - the WebCrawlingContext.

Returns:

true if the link is allowed to be followed, false otherwise

Throws:

WebCrawlerException