org.eclipse.xtext.parser.antlr
Class XtextTokenStream

java.lang.Object
  org.antlr.runtime.CommonTokenStream
      org.eclipse.xtext.parser.antlr.XtextTokenStream

All Implemented Interfaces:

org.antlr.runtime.IntStream, org.antlr.runtime.TokenStream

Direct Known Subclasses:

ObservableXtextTokenStream

public class XtextTokenStream
extends org.antlr.runtime.CommonTokenStream

A token stream that is aware of the current lookahead.

Author:

Jan Köhnlein - Initial contribution and API, Sebastian Zarnekow - Support for dynamic hidden tokens, reworked lookahead algorithm

Nested Class Summary

static interface

XtextTokenStream.HiddenTokens

Field Summary

Fields inherited from class org.antlr.runtime.CommonTokenStream

channel, channelOverrideMap, discardOffChannelTokens, discardSet, lastMarker, p, tokens, tokenSource

Constructor Summary

XtextTokenStream()

XtextTokenStream(org.antlr.runtime.TokenSource tokenSource, int channel)

XtextTokenStream(org.antlr.runtime.TokenSource tokenSource, ITokenDefProvider tokenDefProvider)

Method Summary

void	consume() Move the input pointer to the next incoming token.
protected void	doSetHiddenTokens(java.lang.String... lexerRules)
int	getCurrentLookAhead()
protected int	getFirstMarker()
java.lang.String	getLexerErrorMessage(org.antlr.runtime.Token invalidToken)
void	initCurrentLookAhead(int currentLookAhead)
int	LA(int i) Get int at current input pointer + i ahead where i=1 is next int.
org.antlr.runtime.Token	LT(int k) Same as CommonTokenStream.LT(int) except that we skip hidden tokens even for k == 1.
int	mark() Tell the stream to start buffering if it hasn't already.
void	seek(int index) Set the input cursor to the position indicated by index.
XtextTokenStream.HiddenTokens	setHiddenTokens(java.lang.String... lexerRules)
void	setInitialHiddenTokens(java.lang.String... lexerRules)
protected void	skipHiddenTokens()
java.lang.String	toString(int start, int stop) Return the text of all tokens from start to stop, inclusive.

Methods inherited from class org.antlr.runtime.CommonTokenStream

discardOffChannelTokens, discardTokenType, fillBuffer, get, getSourceName, getTokens, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LB, release, reset, rewind, rewind, setTokenSource, setTokenTypeChannel, size, skipOffTokenChannels, skipOffTokenChannelsReverse, toString, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

XtextTokenStream

public XtextTokenStream()

XtextTokenStream

public XtextTokenStream(org.antlr.runtime.TokenSource tokenSource,
                        int channel)

XtextTokenStream

public XtextTokenStream(org.antlr.runtime.TokenSource tokenSource,
                        ITokenDefProvider tokenDefProvider)

Method Detail

toString

public java.lang.String toString(int start,
                                 int stop)

Description copied from interface: org.antlr.runtime.TokenStream

Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.

Specified by:

toString in interface org.antlr.runtime.TokenStream

Overrides:

toString in class org.antlr.runtime.CommonTokenStream

setHiddenTokens

public XtextTokenStream.HiddenTokens setHiddenTokens(java.lang.String... lexerRules)

setInitialHiddenTokens

public void setInitialHiddenTokens(java.lang.String... lexerRules)

doSetHiddenTokens

protected void doSetHiddenTokens(java.lang.String... lexerRules)

skipHiddenTokens

protected void skipHiddenTokens()

LA

public int LA(int i)

Description copied from interface: org.antlr.runtime.IntStream

Get int at current input pointer + i ahead where i=1 is next int. Negative indexes are allowed. LA(-1) is previous token (token just matched). LA(-i) where i is before first token should yield -1, invalid char / EOF.

Specified by:

LA in interface org.antlr.runtime.IntStream

Overrides:

LA in class org.antlr.runtime.CommonTokenStream

consume

public void consume()

Description copied from class: org.antlr.runtime.CommonTokenStream

Move the input pointer to the next incoming token. The stream must become active with LT(1) available. consume() simply moves the input pointer so that LT(1) points at the next input symbol. Consume at least one token. Walk past any token not on the channel the parser is listening to.

Specified by:

consume in interface org.antlr.runtime.IntStream

Overrides:

consume in class org.antlr.runtime.CommonTokenStream

getFirstMarker

protected int getFirstMarker()

Since:

2.1

mark

public int mark()

Description copied from interface: org.antlr.runtime.IntStream

Tell the stream to start buffering if it hasn't already. Return current input position, index(), or some other marker so that when passed to rewind() you get back to the same spot. rewind(mark()) should not affect the input cursor. The Lexer track line/col info as well as input index so its markers are not pure input indexes. Same for tree node streams.

Specified by:

mark in interface org.antlr.runtime.IntStream

Overrides:

mark in class org.antlr.runtime.CommonTokenStream

seek

public void seek(int index)

Description copied from interface: org.antlr.runtime.IntStream

Set the input cursor to the position indicated by index. This is normally used to seek ahead in the input stream. No buffering is required to do this unless you know your stream will use seek to move backwards such as when backtracking. This is different from rewind in its multi-directional requirement and in that its argument is strictly an input cursor (index). For char streams, seeking forward must update the stream state such as line number. For seeking backwards, you will be presumably backtracking using the mark/rewind mechanism that restores state and so this method does not need to update state when seeking backwards. Currently, this method is only used for efficient backtracking using memoization, but in the future it may be used for incremental parsing. The index is 0..n-1. A seek to position i means that LA(1) will return the ith symbol. So, seeking to 0 means LA(1) will return the first element in the stream.

Specified by:

seek in interface org.antlr.runtime.IntStream

Overrides:

seek in class org.antlr.runtime.CommonTokenStream

LT

public org.antlr.runtime.Token LT(int k)

Same as CommonTokenStream.LT(int) except that we skip hidden tokens even for k == 1.

Specified by:

LT in interface org.antlr.runtime.TokenStream

Overrides:

LT in class org.antlr.runtime.CommonTokenStream

getCurrentLookAhead

public int getCurrentLookAhead()

initCurrentLookAhead

public void initCurrentLookAhead(int currentLookAhead)

getLexerErrorMessage

public java.lang.String getLexerErrorMessage(org.antlr.runtime.Token invalidToken)