org.eclipse.smila.importing.crawler.web.utils

Class UriHelper

java.lang.Object

org.eclipse.smila.importing.crawler.web.utils.UriHelper

public final class UriHelper
extends java.lang.Object

helper class for working with URLs.

Method Summary

Methods

Modifier and Type	Method and Description
static java.lang.String	getDomain(java.lang.String host)
static java.lang.String	getHost(java.lang.String url)
static java.lang.String	getHostAndPort(java.lang.String urlString)
static java.lang.String	getHostAndPort(java.net.URL url)
static java.lang.String	makeAbsolute(java.lang.String baseUriString, java.lang.String uriString)
static java.lang.String	normalizeUrl(java.lang.String context, java.lang.String urlString) URI normalization: make relative URLs absolute wrt. to context (if given) normalize path (host/path/..

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

makeAbsolute

public static java.lang.String makeAbsolute(java.lang.String baseUriString,
                            java.lang.String uriString)
                                     throws java.net.URISyntaxException

Returns:

absolute URI from given URI by using base URI. The base URI is expected to be absolute!

Throws:

java.net.URISyntaxException

getHost

public static java.lang.String getHost(java.lang.String url)
                                throws java.net.MalformedURLException

Returns:

host from given URL string.

Throws:

java.net.MalformedURLException

getHostAndPort

public static java.lang.String getHostAndPort(java.lang.String urlString)
                                       throws java.net.MalformedURLException

Returns:

"host:port" from given URL string.

Throws:

java.net.MalformedURLException

getHostAndPort

public static java.lang.String getHostAndPort(java.net.URL url)

Returns:

"host:port" from given URL.

getDomain

public static java.lang.String getDomain(java.lang.String host)

Returns:

domain from given host string. This is a simple implementation that just cuts off the first part of the host name, if at least two parts remain. This won't work for sth. like 'bbc.co.uk'. (A more sophisticated solution would have to use something like a TLD list, see https://wiki.mozilla.org/TLD_List)

normalizeUrl

public static java.lang.String normalizeUrl(java.lang.String context,
                            java.lang.String urlString)
                                     throws java.net.URISyntaxException

URI normalization:

• make relative URLs absolute wrt. to context (if given)
• normalize path (host/path/../path2 -> host/path2)
• convert scheme and host to lower case (HTTP -> http, WWW.Host.de -> www.host.de)
• remove fragment parts (...#fragment)
• remove default port 80 (host:80 -> host)
• filter out 'opaque' URIs (e.g. mailto:...@...)

Returns:

'null' if urlString couldn't be normalized.

Throws:

java.net.URISyntaxException