ktoken/com.aallam.ktoken/Tokenizer/decode

decode

abstract fun decode(tokens: List<Int>): String

Decodes the given sequence of integers (tokens) back into text based on the underlying encoding scheme.

Return

The decoded text.

Parameters

tokens

The array of integers to be decoded.

abstract fun decode(token: Int): String

Decodes a token into bytes.

NOTE: this will decode all special tokens.

Raises an exception if the token is not in the vocabulary.