A TokenFactory implementation that also serves as a TokenSource, from which some TokenStream may be constructed.

The token index and start index is incremented with each token. In addition, the tokens are stored in a queue, from which they may be taken with nextToken().

Usage of this class is intended to be separated into two phases:

  1. Create tokens with the token() method;
  2. Obtain tokens with the nextToken() method.

I. e., you should only construct a token stream from this token source when you’re done with using it as a token factory. If you mix the two usages, the token source might return an EOF token too early.

no subtypes hierarchy

TokenSourceTokenFactory(Integer initialIndex = ..., Integer initialStartIndex = ...)
  • initialIndex

    The token index of the first token.

  • initialStartIndex

    The start index of the first token.

sourceNameSource Codeshared actual String sourceName
Refines TokenSource.sourceName
Inherited Attributes
Attributes inherited from: Object
hash, string
Attributes inherited from: TokenSource
nextTokenSource Codeshared actual Token nextToken()

Returns the next token, or an EOF token if there is no next token.

Refines TokenSource.nextToken
tokenSource Codeshared actual CommonToken token(String text, Integer type, Integer length)

Creates a CommonToken with the specified text, type, and length.

Inherited Methods
Methods inherited from: Object
Methods inherited from: TokenFactory
Methods inherited from: TokenSource