Class XtextTokenStream

  extended by org.antlr.runtime.CommonTokenStream
      extended by org.eclipse.xtext.parser.antlr.XtextTokenStream
All Implemented Interfaces:
org.antlr.runtime.IntStream, org.antlr.runtime.TokenStream
Direct Known Subclasses:

public class XtextTokenStream
extends org.antlr.runtime.CommonTokenStream

A token stream that is aware of the current lookahead.

Jan Köhnlein - Initial contribution and API, Sebastian Zarnekow - Support for dynamic hidden tokens, reworked lookahead algorithm

Nested Class Summary
static interface XtextTokenStream.HiddenTokens
Field Summary
Fields inherited from class org.antlr.runtime.CommonTokenStream
channel, channelOverrideMap, discardOffChannelTokens, discardSet, lastMarker, p, tokens, tokenSource
Constructor Summary
XtextTokenStream(org.antlr.runtime.TokenSource tokenSource, int channel)
XtextTokenStream(org.antlr.runtime.TokenSource tokenSource, ITokenDefProvider tokenDefProvider)
Method Summary
 void consume()
          Move the input pointer to the next incoming token.
protected  void doSetHiddenTokens(java.lang.String... lexerRules)
 int getCurrentLookAhead()
 java.lang.String getLexerErrorMessage(org.antlr.runtime.Token invalidToken)
 void initCurrentLookAhead(int currentLookAhead)
 int LA(int i)
          Get int at current input pointer + i ahead where i=1 is next int.
 org.antlr.runtime.Token LT(int k)
          Same as CommonTokenStream.LT(int) except that we skip hidden tokens even for k == 1.
 int mark()
          Tell the stream to start buffering if it hasn't already.
 void seek(int index)
          Set the input cursor to the position indicated by index.
 XtextTokenStream.HiddenTokens setHiddenTokens(java.lang.String... lexerRules)
 void setInitialHiddenTokens(java.lang.String... lexerRules)
protected  void skipHiddenTokens()
 java.lang.String toString(int start, int stop)
          Return the text of all tokens from start to stop, inclusive.
Methods inherited from class org.antlr.runtime.CommonTokenStream
discardOffChannelTokens, discardTokenType, fillBuffer, get, getSourceName, getTokens, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LB, release, reset, rewind, rewind, setTokenSource, setTokenTypeChannel, size, skipOffTokenChannels, skipOffTokenChannelsReverse, toString, toString
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail


public XtextTokenStream()


public XtextTokenStream(org.antlr.runtime.TokenSource tokenSource,
                        int channel)


public XtextTokenStream(org.antlr.runtime.TokenSource tokenSource,
                        ITokenDefProvider tokenDefProvider)
Method Detail


public java.lang.String toString(int start,
                                 int stop)
Description copied from interface: org.antlr.runtime.TokenStream
Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.

Specified by:
toString in interface org.antlr.runtime.TokenStream
toString in class org.antlr.runtime.CommonTokenStream


public XtextTokenStream.HiddenTokens setHiddenTokens(java.lang.String... lexerRules)


public void setInitialHiddenTokens(java.lang.String... lexerRules)


protected void doSetHiddenTokens(java.lang.String... lexerRules)


protected void skipHiddenTokens()


public int LA(int i)
Description copied from interface: org.antlr.runtime.IntStream
Get int at current input pointer + i ahead where i=1 is next int. Negative indexes are allowed. LA(-1) is previous token (token just matched). LA(-i) where i is before first token should yield -1, invalid char / EOF.

Specified by:
LA in interface org.antlr.runtime.IntStream
LA in class org.antlr.runtime.CommonTokenStream


public void consume()
Description copied from class: org.antlr.runtime.CommonTokenStream
Move the input pointer to the next incoming token. The stream must become active with LT(1) available. consume() simply moves the input pointer so that LT(1) points at the next input symbol. Consume at least one token. Walk past any token not on the channel the parser is listening to.

Specified by:
consume in interface org.antlr.runtime.IntStream
consume in class org.antlr.runtime.CommonTokenStream


public int mark()
Description copied from interface: org.antlr.runtime.IntStream
Tell the stream to start buffering if it hasn't already. Return current input position, index(), or some other marker so that when passed to rewind() you get back to the same spot. rewind(mark()) should not affect the input cursor. The Lexer track line/col info as well as input index so its markers are not pure input indexes. Same for tree node streams.

Specified by:
mark in interface org.antlr.runtime.IntStream
mark in class org.antlr.runtime.CommonTokenStream


public void seek(int index)
Description copied from interface: org.antlr.runtime.IntStream
Set the input cursor to the position indicated by index. This is normally used to seek ahead in the input stream. No buffering is required to do this unless you know your stream will use seek to move backwards such as when backtracking. This is different from rewind in its multi-directional requirement and in that its argument is strictly an input cursor (index). For char streams, seeking forward must update the stream state such as line number. For seeking backwards, you will be presumably backtracking using the mark/rewind mechanism that restores state and so this method does not need to update state when seeking backwards. Currently, this method is only used for efficient backtracking using memoization, but in the future it may be used for incremental parsing. The index is 0..n-1. A seek to position i means that LA(1) will return the ith symbol. So, seeking to 0 means LA(1) will return the first element in the stream.

Specified by:
seek in interface org.antlr.runtime.IntStream
seek in class org.antlr.runtime.CommonTokenStream


public org.antlr.runtime.Token LT(int k)
Same as CommonTokenStream.LT(int) except that we skip hidden tokens even for k == 1.

Specified by:
LT in interface org.antlr.runtime.TokenStream
LT in class org.antlr.runtime.CommonTokenStream


public int getCurrentLookAhead()


public void initCurrentLookAhead(int currentLookAhead)


public java.lang.String getLexerErrorMessage(org.antlr.runtime.Token invalidToken)