LEXER_BODY_FUNCTION

The function to run (as the base call of a fiber) to generate some tokens from the source string and position. The function should produce a tuple of potential A_Tokens at this position, as produced by this lexer. Each token may be seeded with the potential tokens that follow it. Since each token also records the LexingState after it, there's no need to produce that separately.

Properties

Link copied to clipboard
val name: String
Link copied to clipboard
val ordinal: Int