Tokenizer
latent_space.services.tokenizer
¶
TokenizerService
¶
Tokenizes text with a GPT-2 byte-level BPE encoding.
Exposes the real token ids and a lossless per-token character mapping so callers can render both the integer ids and readable text without ever fabricating a token GPT-2 does not have.
tokenize(text)
¶
Encode text and return its tokens, ids, and counts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
Arbitrary text; special-token strings are treated as ordinary text. |
required |
Returns:
| Type | Description |
|---|---|
TokenizeResponse
|
The per-token spans and the token, word, and character counts. |