to.etc.lexer

Class TextTokenizerBase

java.lang.Object

to.etc.lexer.TextReaderBase

to.etc.lexer.ReaderScannerBase

to.etc.lexer.TextTokenizerBase

public abstract class TextTokenizerBase
extends ReaderScannerBase

A tokenizing lexer which passes information into LexerToken instances, allowing tokens to be pushed back where needed. This does no token management (responsibility of caller) because that performs way better when using token instances.

Author:

Frits Jalvingh Created on Oct 28, 2009

Field Summary

Fields inherited from class to.etc.lexer.ReaderScannerBase

T_BASE_LAST, T_COMMENT, T_EOF, T_IDENT, T_IPADDR, T_NUMBER, T_STRING

Constructor Summary

Constructors

Constructor and Description
TextTokenizerBase(Object source, Reader r)

Method Summary

Modifier and Type	Method and Description
int	nextToken(LexerToken t) Checks if the token queue contains tokens, and if so returns the topmost one.
protected abstract int	nextTokenPrim() Handles actual token generation using the reader's globals.
LexerToken	popToken()
void	pushToken(LexerToken t) Push the specified token into the token queue.
void	pushTokenCopy(LexerToken t)
int	tokenStackSize()

Methods inherited from class to.etc.lexer.ReaderScannerBase

error, getSourceLocation, getTokenColumn, getTokenLine, isAllowNewlineInString, isIdChar, isIdStart, scanIdentifier, scanNumber, scanSimpleString, scanUndottedNumber, setAllowNewlineInString, skipWs, skipWsNoNL, startToken, tokenString

Methods inherited from class to.etc.lexer.TextReaderBase

accept, accept, append, append, clearCopy, copy, copy, getCopied, getCurrentColumn, getCurrentLine, getSource, LA, LA, sb

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextTokenizerBase

public TextTokenizerBase(Object source,
                         Reader r)

Method Detail

nextTokenPrim

protected abstract int nextTokenPrim()
                              throws Exception

Handles actual token generation using the reader's globals.

Parameters:

t -

Returns:

Throws:

Exception

pushToken

public void pushToken(LexerToken t)

Push the specified token into the token queue. The token becomes OWNED by this - it should not be re-used.

Parameters:

t -

pushTokenCopy

public void pushTokenCopy(LexerToken t)

popToken

public LexerToken popToken()

tokenStackSize

public int tokenStackSize()

nextToken

public int nextToken(LexerToken t)
              throws Exception

Checks if the token queue contains tokens, and if so returns the topmost one. If not this scans input for the next token.

Parameters:

t -

Returns:

Throws:

Exception