Token
Token
📖 Definition
Token is the basic unit for processing text in large language models. Models split input text into a series of tokens for processing, where each token may be a word, subword, or character. Token count directly affects API call costs and response time.
🔗 How Higress Uses This
Higress AI Gateway provides token-level traffic management, supporting quota management and rate limiting based on token consumption, helping enterprises precisely control AI call costs.
💡 Examples
- 1 "Hello World" in English is typically split into 2 tokens
- 2 Chinese tokenization is usually done by character or word group
- 3 GPT-4's context window supports 128K tokens
⚙️ Configuration Example
YAML
# Higress Token Rate Limiting Configuration Example
rule:
limit_by_token:
tokens_per_minute: 100000
tokens_per_day: 1000000🔄 Related Terms
❓ FAQ
What is Token?
Token is the basic unit for processing text in large language models. Models split input text into a series of tokens for processing, where each token may be a word, subword, or character. Token count directly affects API call costs and response time.
How does Higress support Token?
Higress AI Gateway provides token-level traffic management, supporting quota management and rate limiting based on token consumption, helping enterprises precisely control AI call costs.
Learn More About Higress
Explore more Higress features and best practices