ktoken/com.aallam.ktoken/Tokenizer/Companion/of

of

suspend fun of(encoding: Encoding, loader: BpeLoader = defaultPbeLoader()): Tokenizer

Asynchronously obtains an instance of Tokenizer with the specified encoding scheme.

Return

An instance of Tokenizer with the specified encoding.

Parameters

encoding

The encoding scheme to be used.

loader

The loader to be used for obtaining the encoding scheme.

suspend fun of(model: String, loader: BpeLoader = defaultPbeLoader()): Tokenizer

Asynchronously obtains an instance of Tokenizer for the specified model name.

Return

An instance of Tokenizer with the encoding scheme for the specified model.

Parameters

model

The name of the model whose encoding scheme is to be used.

loader

The loader to be used for obtaining the encoding scheme.