Token
Token
D2
In OpenAI’s documented text processing, a token is a unit into which text is divided for a model to process or generate.
Review status: 2026-11-26
Technical explanation
In OpenAI’s documentation, a token can represent a character, part of a word, a whole word, punctuation, or spacing depending on encoding and context.
Conceptual boundaries
In OpenAI’s documentation, a token is not necessarily a word, and token counts can differ for the same text across encodings and languages.
Provider-neutral example
Before submitting a long request, a team can count the tokens produced by the target model’s tokenizer instead of estimating only its word count.
Limitations
Token examples and counts are encoding-specific, so a count from one model does not automatically apply to another model.
Related concepts
Atomic claims and evidence
1.1OpenAI Help Center, Understanding and counting tokens
- Source
- OpenAI Help Center, Understanding and counting tokens
- Source role
- Authoritative source
- Exact locator
- Overview: token units and possible text segments
- Supported claim
- Tokens are units that a model uses to process text, and a token can represent a character, part of a word, a whole word, or punctuation.
- Last verification
- Review due
- Scope limitation
- This is official technical documentation for its models’ text processing and does not prescribe one tokenizer for all language models.
2.1OpenAI Help Center, Understanding and counting tokens
- Source
- OpenAI Help Center, Understanding and counting tokens
- Source role
- Authoritative source
- Exact locator
- Overview and Understand how text becomes tokens: count variability
- Supported claim
- For OpenAI’s documented text models, token count is not the same as word count and can vary with the model, encoding, and language.
- Last verification
- Review due
- Scope limitation
- This is provider-bounded documentation and does not establish a fixed conversion ratio or universal tokenizer behavior.