Tokenizer (WoolCore 1.0.1 API)

java.lang.Object
- nl.rrd.wool.expressions.Tokenizer

```
public class Tokenizer
extends Object
```
A tokenizer can read tokens from text input. It is used by the ExpressionParser. A token is an atomic element of an expression.
After construction, you can read tokens with readToken(). If you construct the tokenizer with a LineColumnNumberReader, then you can rewind the input after each call of readToken(), regardless of whether it returns a token or null or a parse error. Rewinding can be useful if the tokens are embedded inside other text input.

Constructor Summary

Constructors
Constructor Description

Tokenizer(Reader reader)

Tokenizer(String input)

Tokenizer(LineColumnNumberReader reader)

Method Summary

Modifier and Type	Method	Description
`void`	`close()`	Closes the tokenizer and the input reader.
`int`	`getColNum()`	Returns the current column number.
`int`	`getLineNum()`	Returns the current line number.
`LineColumnNumberReader`	`getReader()`	Returns the reader used by this tokenizer.
`Token`	`readToken()`	Tries to read the next token.
`void`	`rewind()`	Rewinds the `LineColumnNumberReader` so it is reset to the same position as before the last call of `readToken()`.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Tokenizer
```
public Tokenizer(String input)
```
  - Tokenizer
```
public Tokenizer(Reader reader)
```
  - Tokenizer
```
public Tokenizer(LineColumnNumberReader reader)
```
- Method Detail
  - getReader
```
public LineColumnNumberReader getReader()
```
    Returns the reader used by this tokenizer.
    
    Returns:
    
    the reader
  - close
```
public void close()
           throws IOException
```
    Closes the tokenizer and the input reader.
    
    Throws:
    
    IOException - if the reader can't be closed
  - readToken
```
public Token readToken()
                throws LineNumberParseException,
                       IOException
```
    Tries to read the next token. If the end of input is reached, this method returns null.
    
    Returns:
    
    the token or null
    
    Throws:
    
    LineNumberParseException - if a parse error occurs
    
    IOException - if a reading error occurs
  - getLineNum
```
public int getLineNum()
```
    Returns the current line number.
    
    Returns:
    
    the current line number
  - getColNum
```
public int getColNum()
```
    Returns the current column number.
    
    Returns:
    
    the current column number
  - rewind
```
public void rewind()
            throws IOException
```
    Rewinds the LineColumnNumberReader so it is reset to the same position as before the last call of readToken().
    
    Throws:
    
    IOException - if an I/O error occurs