Entity
Tokenizer
A component that converts text into numerical tokens for AI model processing, with different strategies affecting context window comparisons across platforms.
术语属性
- purpose：text preprocessing
- impact：context window efficiency
- variability：model-specific implementation