org.eclipse.xtext.parser.antlr
Class AbstractSplittingTokenSource

java.lang.Object
  org.eclipse.xtext.parser.antlr.AbstractSplittingTokenSource

All Implemented Interfaces:

org.antlr.runtime.TokenSource

public abstract class AbstractSplittingTokenSource
extends java.lang.Object
implements org.antlr.runtime.TokenSource

Delegating token source that is capable of splitting a given token into other tokens. This allows clients to implement sophisticated semantics when dealing with certain token types in different use cases, e.g. a token source that will provide tokens for a python-like whitespace aware language.

Author:

Sebastian Zarnekow - Initial contribution and API

Field Summary

static java.lang.String

LEXER_RULE_PREFIX

Constructor Summary

protected

AbstractSplittingTokenSource()

Method Summary

protected org.antlr.runtime.CommonToken	createToken(org.antlr.runtime.CommonToken prototype, java.lang.String text, java.lang.Integer charPosInLine, java.lang.Integer channel, java.lang.Integer start, java.lang.Integer stop, java.lang.Integer type) Create a new token from the given prototype.
protected abstract void	doSplitToken(org.antlr.runtime.Token splitMe, ITokenAcceptor acceptor)
org.antlr.runtime.TokenSource	getDelegate()
java.lang.String	getLexerRuleName(java.lang.String antlrTokenDef)
protected java.lang.String	getRuleName(org.antlr.runtime.Token token)
java.lang.String	getSourceName() Where are you getting tokens from?
protected int	getTokenType(java.lang.String ruleNameWithRulePrefix)
void	initializeTokenDefsFrom(ITokenDefProvider provider)
org.antlr.runtime.Token	nextToken() Return a Token object from your input stream (usually a CharStream).
void	setDelegate(org.antlr.runtime.TokenSource delegate)
protected abstract boolean	shouldSplitToken(org.antlr.runtime.Token candidate)
protected void	splitToken(org.antlr.runtime.Token splitMe, ITokenAcceptor acceptor)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LEXER_RULE_PREFIX

public static final java.lang.String LEXER_RULE_PREFIX

See Also:

Constant Field Values

Constructor Detail

AbstractSplittingTokenSource

protected AbstractSplittingTokenSource()

Method Detail

setDelegate

public void setDelegate(org.antlr.runtime.TokenSource delegate)

getDelegate

public org.antlr.runtime.TokenSource getDelegate()

nextToken

public org.antlr.runtime.Token nextToken()

Description copied from interface: org.antlr.runtime.TokenSource

Return a Token object from your input stream (usually a CharStream). Do not fail/return upon lexing error; keep chewing on the characters until you get a good one; errors are not passed through to the parser.

Specified by:

nextToken in interface org.antlr.runtime.TokenSource

getSourceName

public java.lang.String getSourceName()

Description copied from interface: org.antlr.runtime.TokenSource

Where are you getting tokens from? normally the implication will simply ask lexers input stream.

Specified by:

getSourceName in interface org.antlr.runtime.TokenSource

getRuleName

protected java.lang.String getRuleName(org.antlr.runtime.Token token)

getTokenType

protected int getTokenType(java.lang.String ruleNameWithRulePrefix)

getLexerRuleName

public java.lang.String getLexerRuleName(java.lang.String antlrTokenDef)

initializeTokenDefsFrom

@Inject
public void initializeTokenDefsFrom(ITokenDefProvider provider)

splitToken

protected void splitToken(org.antlr.runtime.Token splitMe,
                          ITokenAcceptor acceptor)

shouldSplitToken

protected abstract boolean shouldSplitToken(org.antlr.runtime.Token candidate)

doSplitToken

protected abstract void doSplitToken(org.antlr.runtime.Token splitMe,
                                     ITokenAcceptor acceptor)

createToken

protected org.antlr.runtime.CommonToken createToken(org.antlr.runtime.CommonToken prototype,
                                                    java.lang.String text,
                                                    java.lang.Integer charPosInLine,
                                                    java.lang.Integer channel,
                                                    java.lang.Integer start,
                                                    java.lang.Integer stop,
                                                    java.lang.Integer type)

Create a new token from the given prototype. Any argument besides the prototype is optional and will be ignored if its value is null.