Text and source-code token counting
Technical reference for browser-based text and source-code token counting, OpenAI BPE vocabularies, UTF-8 projections, and result accuracy labels.
Text remains in the browser. The calculator selects either a local tokenizer vocabulary or a disclosed deterministic provider-family projection, then reports the method next to the result.
Text pipeline
- Decode pasted or file text into a JavaScript Unicode string.
- Encode with the selected local BPE vocabulary, or measure UTF-8 bytes for the provider projection.
- Add extracted text across supported files.
- Apply the selected model's price and context limit.
Exact local BPE path
OpenAI-family raw text uses js-tiktoken with the model registry's configured encoding, including o200k_base where applicable. “Exact” describes segmentation of that raw string under that vocabulary. It does not include chat framing, tools, hosted file processing, or server instructions.
Provider-calibrated UTF-8 projection
When a provider does not distribute an equivalent offline production tokenizer, the calculator divides UTF-8 byte length by a versioned provider-family coefficient. This path is deterministic and reproducible, but it is a projection. Claude, Gemini, and DeepSeek API usage remains authoritative.
Why code behaves differently
Identifiers, punctuation, whitespace, escaped strings, hashes, and minified content change segmentation density. Count the real artifact rather than converting characters or words with a universal ratio. Supported source files enter the same text pipeline without remote upload.
Practical counting examples
| Input | What changes token density | Recommended check |
|---|---|---|
| Natural-language prompt | Language, punctuation, and repeated context | Count the final system and user text together. |
| JSON or tool schema | Property names, braces, escaping, and examples | Measure the exact serialized payload. |
| Source code | Identifiers, indentation, comments, and minification | Upload the real source file instead of estimating from lines. |
| Logs or CSV | Repeated timestamps, IDs, delimiters, and long values | Count a representative production sample. |
Exact count versus provider bill
An exact local BPE result describes segmentation of the supplied string. Provider usage can still include hidden instructions, message wrappers, tool definitions, cached prefixes, reasoning tokens, or generated output. Treat the local count as a reproducible preflight measurement and reconcile production billing with the API response.