Token Limits Break Everything
| Status | Duration | Type | Tools |
|---|---|---|---|
| Observed | — | Note | — |
Observation
While testing an AI workflow, we noticed that the model consistently fails when given inputs with special characters. Turns out it was hitting token limits we didn't know existed.
The failures were silent—no error messages, just truncated outputs that looked correct but were missing critical data. We only caught it because a customer complained that extracted phone numbers were incomplete.
Context
We were processing customer data that included emoji, Unicode characters, and code snippets. The tokenizer was splitting these into way more tokens than expected, silently truncating inputs.
For example: "🎉 Success!" = 5 tokens (not 2 words). A single emoji can be 3-4 tokens depending on the model. We were assuming 1 token ≈ 1 word, which is wrong for non-English text.
What It Means
Always validate input length before sending to the model. Don't trust character count—use the actual tokenizer to estimate token count and warn when you're close to limits.
We added a pre-processing step that uses tiktoken (OpenAI's tokenizer library) to count tokens on every input. If it's >75% of the context window, we split it or reject it with a clear error message.
Action Item
Add token counting to your input validation. Here's the snippet we use:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
token_count = len(enc.encode(input_text))
if token_count > 6000: raise ValueError("Input too long")