ktoken/com.aallam.ktoken.encoding/P50KBase

P50KBase

data class P50KBase(val file: String = "p50k_base.tiktoken") : Encoding

Default configuration of p50k_base encoding.

Constructors

Link copied to clipboard
constructor(file: String = "p50k_base.tiktoken")

Properties

Link copied to clipboard
open override val file: String

The encoding file path.

Functions

Link copied to clipboard
open override fun encodingConfig(ranks: Map<ByteString, Int>): EncodingConfig

Retrieve token encoding utilizing the provided BpeLoader.