encodeSingleToken

Encodes text corresponding to a single token to its token value.

NOTE: this will encode all special tokens.

Raises an exception if the token is not in the vocabulary.

>>> tokenizer.encodeSingleToken("hello")
31373