new
fun newToken( string: A_String, start: Int, lineNumber: Int, tokenType: TokenDescriptor.TokenType, generatingLexer: A_Lexer): A_Token
Content copied to clipboard
Create and initialize a new A_Token. The NEXT_LEXING_STATE_POJO is initially set to nil. For a token constructed by a lexer body, this pojo is updated automatically by the lexical scanning machinery to wrap a new LexingState. That machinery also sets up the new scanning position, the new line number, and the list of LexingState.allTokens.
Return
The new token.
Parameters
string
The token text.
start
The token's starting character position in the file.
line
The line number on which the token occurred.
token
The type of token to create.
generating
The A_Lexer responsible for creating this token, or nil if the token was not constructed by a lexer.