from
Builds and returns an instance of Tokenizer based on the specified EncodingConfig.
Return
An instance of Tokenizer.
Parameters
config
The EncodingConfig object representing the encoding scheme to be used.
Builds and returns an instance of Tokenizer based on the specified EncodingConfig.
An instance of Tokenizer.
The EncodingConfig object representing the encoding scheme to be used.