| Class | Description |
|---|---|
| DeltaHash |
helper class for calculating the delta hash attribute for records produced by the web-crawler.
|
| RobotsTxt |
parses a robots.txt file and checks if paths are allowed.
|
| UriHelper |
helper class for working with URLs.
|
| WebCrawlerConfiguration |
Configuration for the web crawler.
|