ktoken/com.aallam.ktoken/EncodingConfig

EncodingConfig

class EncodingConfig(val pattern: Regex, val mergeableRanks: Map<ByteString, Int>, val specialTokens: Map<ByteString, Int>, val explicitNVocab: Int? = null)

Manages configurations for token encoding, providing the settings and mappings needed to perform byte pair encoding (BPE) and handle special tokens.

Constructors

EncodingConfig

constructor(pattern: Regex, mergeableRanks: Map<ByteString, Int>, specialTokens: Map<ByteString, Int>, explicitNVocab: Int? = null)

Types

Companion

object Companion

Properties

explicitNVocab

val explicitNVocab: Int? = null

The number of tokens in the vocabulary. If provided, it is checked that the number of mergeable tokens and special tokens is equal to this number.

mergeableRanks

val mergeableRanks: Map<ByteString, Int>

A dictionary mapping mergeable token bytes to their ranks. The ranks must correspond to merge priority.

pattern

val pattern: Regex

A regex pattern string that is used to split the input text.

specialTokens

val specialTokens: Map<ByteString, Int>

A dictionary mapping special token strings to their token values.