Companion

Functions

Builds and returns an instance of Tokenizer based on the specified EncodingConfig.

suspend fun of(encoding: Encoding, loader: BpeLoader = defaultPbeLoader()): Tokenizer

Asynchronously obtains an instance of Tokenizer with the specified encoding scheme.

suspend fun of(model: String, loader: BpeLoader = defaultPbeLoader()): Tokenizer

Asynchronously obtains an instance of Tokenizer for the specified model name.