ktoken
common
ktoken
/
com.aallam.ktoken.encoding
/
R50KBase
/
encodingConfig
encoding
Config
open
override
fun
encodingConfig
(
ranks
:
Map
<
ByteString
,
Int
>
)
:
EncodingConfig
Retrieve token encoding utilizing the provided BpeLoader.
Parameters
ranks
byte pair encoding ranks.