avail/avail.compiler.scanning/LexingState

LexingState

class LexingState( val compilationContext: CompilationContext, val position: Int, val lineNumber: Int, val allTokens: List<A_Token>)

LexingState instances represent the lexing state between tokens. They bind together consecutive tokens into a directed acyclic graph, allowing the AvailCompiler to process all possible paths (lexings), typically in aggregate.

Author

Mark van Gulik

Parameters

compilationContext

The CompilationContext in which compilation is occurring.

position

The one-based position in the source.

lineNumber

The one-based line number of this position in the source.

allTokens

The immutable list of A_Tokens that have been parsed up to this position, starting at the current top-level statement, but including any leading whitespace and comment tokens.

Constructors

LexingState

fun LexingState( compilationContext: CompilationContext, position: Int, lineNumber: Int, allTokens: List<A_Token>)

Construct a new immutable LexingState. It starts in a state where the potential tokens at this position have not yet been computed, and no actions have been queued to run against that eventual list of tokens.

Types

Companion

object Companion

Functions

expected

fun expected(level: CompilerDiagnostics.ParseNotificationLevel, describer: Describer)

Record an expectation at the current parse position. The expectations captured at the rightmost few reached parse positions constitute the error message in case the parse fails.

fun expected(level: CompilerDiagnostics.ParseNotificationLevel, aString: String)

Record an indication of what was expected at this parse position.

fun expected( level: CompilerDiagnostics.ParseNotificationLevel, values: List<A_BasicObject>, transformer: Function<List<String>, String>)

Record an expectation at the current parse position. The expectations captured at the rightmost parse position constitute the error message in case the parse fails.

setFiberContinuationsTrackingWork

fun setFiberContinuationsTrackingWork( fiber: A_Fiber, onSuccess: (AvailObject) -> Unit, onFailure: (Throwable) -> Unit)

Set up the given fiber to eventually invoke either the onSuccess or the onFailure continuation, but not both. However, immediately record the fact that we're expecting one of these to be eventually invoked, and wrap the continuations with code that will invoke getNoMoreWorkUnits when the number of outstanding tasks reaches zero.

styleAllTokensThen

fun styleAllTokensThen(then: () -> Unit)

Apply styling to each token in allTokens, which includes all whitespace and comment tokens as well. The token knows its A_Lexer, which is in an A_Method, which can specify an A_Styler in a module-scoped way.

withTokensDo

@Synchronized

fun withTokensDo(newAction: (List<A_Token>) -> Unit)

Eventually invoke the given function with the list of tokens at this position. If necessary, launch fibers to run lexers, invoking the continuation only when all possible next tokens have been computed.

workUnitDo

fun workUnitDo(continuation: () -> Unit)

Eventually invoke the given 0-argument function. Track it as an outstanding action, ensuring CompilationContext.noMoreWorkUnits is invoked only when all such queued actions have completed.

Properties

allTokens

val allTokens: List<A_Token>

Every token, including whitespace, that has been parsed up to this point, from the start of the current top-level statement. This should also include the initial leading whitespace, as we like to put related comments before statements.

compilationContext

val compilationContext: CompilationContext

The compilation context for which this is a state of lexing.

knownToBeComputedTokensOrNull

@get:Synchronized

val knownToBeComputedTokensOrNull: List<A_Token>?

If the A_Tokens that start at this location have been computed, answer that List, otherwise answer null. The list is copied to ensure it isn't modified by the client or other active lexers.

lineNumber

val lineNumber: Int

The one-based line number at which this state occurs in the source.

position

val position: Int

The position represented by this LexingState. In particular, it's the (one-based) start of the current token within the source.