Skip to content

latent-space docs

Tokenizer

latent-space docs

Home
Updating from the Template
API Reference
API Reference
- App
- Constants
- Api
  Api
  - Chat
  - Content
  - Dependencies
  - Monitoring
  - Tokenize
- Core
  Core
  - Settings
  - Version
- Models
  Models
  - Content
  - Monitoring
  - Tokenizer Tokenizer
    Table of contents
    
    tokenizer
    
    TokenSpan
- Services
  Services

Tokenizer

`latent_space.models.tokenizer` ¶

`TokenSpan` ¶

Bases: BaseModel

One GPT-2 token: its id and the characters attributed to it.

text is empty for a token that holds only continuation bytes of a multi-byte character (e.g. part of an emoji); that character is attributed to the token holding its first byte, keeping the joined text lossless.