LuAITools.com
提交工具
🔤AI
The smallest unit of AI language

Token

A token is the smallest unit of language an AI works with. Models don't read letters — they chop text into tokens first, which is why they bill by usage and cap your length.

What is a token?

You'd assume ChatGPT is reading your English, but it doesn't actually "know" letters. Every sentence you send gets chopped into little fragments first — those fragments are tokens. A short English word is usually one token; a longer word might be two or three.
The model works with these tokens, not with words themselves.

Why it matters

It decides cost and length
API pricing, free-tier limits, the maximum context length — all of it is tokens doing the accounting. Models charge per token, and each one has a hard cap on how many it can take at once.
It's the reason some languages are "cheaper"
Languages tokenize differently. A sentence in one language might use fewer tokens than the same idea in another, which quietly changes the bill.

A quick example

The phrase "I love apples" might become three tokens: "I", "love", "apples". "I really love apples" adds one more. The finer the split, the more precisely the model reads — and the more it costs.

Bottom line: tokens are the AI's unit of measurement. Every dollar you spend, every inch of context you occupy, is being quietly counted in tokens.

Comments