How it works
LLMs do not process text as characters or words, they use tokens. A tokenizer splits input text into tokens based on patterns learned from training data. Common words like "the" or "hello" are single tokens, while uncommon words are split into multiple tokens ("tokenization" might become "token" + "ization").
Tokens matter for three practical reasons:
- Pricing: LLM APIs charge per token (input + output). More tokens = higher cost.
- Context window: Each model has a maximum token limit for the combined input and output. Exceeding it means truncating context.
- Latency: More output tokens = longer response time, since LLMs generate one token at a time.
For a typical customer support interaction: the system prompt uses 200-500 tokens, RAG context uses 500-2,000 tokens, the customer question uses 20-100 tokens, and the AI response uses 100-500 tokens.
Why it matters
Understanding tokens is essential for managing AI chatbot costs and performance. Token usage directly determines your monthly LLM costs, response speed, and how much context the AI can consider when generating answers. Over-engineering prompts or including excessive RAG context wastes tokens and money without improving answer quality.
How Chatsy uses it
Chatsy manages token usage automatically by optimizing system prompts, intelligently selecting the most relevant RAG passages (rather than stuffing the context), and managing conversation history to stay within model limits. Usage-based pricing on Chatsy is calculated from token consumption, making efficient token management a direct cost savings.
Chatsy
What else should I know about Token?
LLM pricing is per-token, making token efficiency directly tied to cost management
Key takeawaysTest this in your own agent
Add your help content, ask the questions on this page, and check every answer before you publish.
Start freeChatsy
Example scenarios
- 01
Token cost calculation for a support conversation
A typical AI support conversation uses: 400 tokens (system prompt) + 1,200 tokens (RAG context) + 50 tokens (customer question) + 200 tokens (AI response) = 1,850 tokens. At GPT-5 pricing, this costs approximately $0.005 per conversation, enabling thousands of AI conversations for dollars, not hundreds.
- 02
Context window management in long conversations
A customer has a 15-message troubleshooting conversation. The full history exceeds the context window, so the system uses a sliding window, keeping the system prompt, the latest RAG context, and the most recent 8 messages while summarizing earlier messages. This maintains conversational continuity within token limits.
- 03
Token optimization for cost reduction
A high-volume support team reduces their AI costs by 40% by: shortening the system prompt from 800 to 300 tokens, limiting RAG context to the top 3 passages instead of 10, and setting a maximum response length of 200 tokens for simple FAQ answers.
Key takeaways
- Tokens are the basic text units LLMs process, roughly 3/4 of a word in English
- LLM pricing is per-token, making token efficiency directly tied to cost management
- Context windows limit total tokens (input + output), requiring careful management for long conversations
- A typical support interaction uses 1,500-3,000 tokens total, costing $0.003-$0.01 depending on the model
- Token optimization (efficient prompts, selective RAG context) reduces costs; check that answer quality holds