Token

Language models

The token is the unit of text a language model works with: a short word, a word fragment, a punctuation mark. Text is split into tokens before processing; API billing, context window size and generation speed are all counted in tokens.

In practice at Gensai

Gensai sizes its architectures in tokens: document chunking for RAG, control of API costs and latency.

Same theme

LLM (Large Language Model)SLM (Small Language Model)DLLM (Diffusion Large Language Model)Hallucination (AI)Prompt chaining Pre-trained model Foundation model Generative AI Transformer Context window Prompt Prompt engineering Context (AI)Memory (AI)Fine-tuning