PDF, DOCX, and data-file processing
Documentation for local PDF and DOCX text extraction, source-code and data-file token counting, supported formats, privacy boundaries, and extraction limits.
Files are processed in the active browser session. TokenCalculator.dev extracts readable content locally and passes only the resulting text to the selected model-counting method.
Supported input classes
Direct text decoding covers TXT, Markdown, CSV, JSON, HTML, XML, YAML, TOML, SQL, shell scripts, and common source-code extensions. Rich-document extraction supports readable PDF and DOCX content. The per-file limit is 12 MB.
PDF extraction
The local parser reconstructs strings from the PDF text layer. Reading order can differ in multi-column layouts, tables, headers, and positioned diagrams. Raster-only scanned pages contain pixels rather than extractable text; OCR is not performed and the file must not receive a fabricated text count.
Pages cost tokens, not only text
Extracted text is not the whole bill. Every provider that accepts a PDF also sends each page to the model as an image, and that page image is charged on top of the words. Gemini publishes the rate—560 tokens a page on Gemini 3 by default, 258 on Gemini 2.x—so the calculator adds it as a separate line. OpenAI, Anthropic, and xAI accept PDFs without publishing a per-page figure, so the calculator counts their text exactly and flags the page images as an unpriced extra. See audio, video, and PDF tokens for the full rate table.
DOCX extraction
DOCX is an Open Packaging Convention archive containing XML parts. The browser extracts readable paragraph content. Comments, revisions, embedded objects, headers, footers, and provider-specific conversion may produce a different server-side representation.
Accuracy and privacy boundaries
- No selected file content is sent to the TokenCalculator.dev calculation server.
- Extraction output—not file bytes—is tokenized.
- A provider's own upload or retrieval API may add framing, OCR, chunks, and citations.
- Validate production usage against response metadata.
Format behavior at a glance
| File class | Local method | Important limitation |
|---|---|---|
| TXT, code, CSV, JSON | UTF-8 text decoding | Binary encodings and malformed text may not decode meaningfully. |
| PDF with text layer | PDF.js text extraction | Columns, tables, and positioned text can change reading order. |
| Scanned PDF | No OCR | Images of text cannot receive a trustworthy text-token count. |
| DOCX | Mammoth paragraph extraction | Headers, comments, revisions, and embedded objects may differ downstream. |
How to get a reliable file estimate
- Use the same document revision that will enter the AI workflow.
- Review extracted character and word totals for obvious empty or scrambled output.
- Add any surrounding prompt instructions before comparing models.
- For production uploads, compare the estimate with provider usage metadata because hosted file APIs may add OCR, chunking, or retrieval framing.