Try Chat Free Docs Pricing AI Analysis Repo Insights Free Tools About Blog
Sign in Get started free
< Back to blog
Research Velona Team ·29 October 2026 ·15 min read

Does AI Cost More in Hindi? We Measured the Token Tax for 9 Indian Languages Across 7 Tokenizers

Short answer: yes. AI models charge per token, and Indian languages need more tokens than English to say the same thing. In our measurement, the same document needed 1.65x as many tokens in Hindi, 2.18x in Kannada, 2.51x in Tamil and 2.80x in Punjabi on the tokenizer used by OpenAI's GPT-5 family. Across nine Indian languages, the average was 1.99x. On some other tokenizers, including Qwen 3 and Llama 3, the same text cost 6x to 10x as many tokens as English.

That multiplier hits you twice. You pay more per message, and each message uses up more of the model's context window.

The good news is that the tax varies hugely by model. The tokenizer that Google's tooling uses for Gemini models had the lowest average tax in our test, at 1.77x, and the gap between the best and worst tokenizer was more than 4x. Picking the right model for Indian-language work is one of the cheapest optimisations available.

This post explains how we measured it, the full results, what it means in rupees and how to reduce it.

Token tax at a glance

The table shows how many tokens each language needs for the same text, relative to English (English = 1.00). Lower is cheaper. The text is the full Universal Declaration of Human Rights in each language.

LanguageGPT-5 family (o200k)Gemini (Gemma 3 tokenizer)Mistral (Tekken)DeepSeek V3Llama 3Qwen 3GPT-4 (cl100k, older)
Hindi1.651.371.903.122.995.315.66
Bengali1.661.141.952.055.945.175.98
Urdu1.701.581.802.393.073.364.55
Gujarati1.711.822.954.138.467.408.46
Malayalam1.801.652.634.218.657.138.74
Marathi1.901.452.263.333.175.425.89
Kannada2.182.132.483.919.447.429.44
Tamil2.511.892.803.819.747.609.74
Punjabi2.802.903.234.948.758.298.84
Average (9 languages)1.991.772.443.546.696.347.48

Three patterns stand out:

How we measured it

We wanted a fair, repeatable test that anyone can rerun, so we used public text and public tokenizers.

The text

We used the Universal Declaration of Human Rights (UDHR), a document translated into hundreds of languages. We took the Unicode versions from the udhr2 corpus distributed with NLTK, for English and nine Indian languages: Hindi, Bengali, Urdu, Gujarati, Malayalam, Marathi, Kannada, Tamil and Punjabi.

Using one document in every language means we compare the same meaning, not just the same number of words. We collapsed runs of whitespace into single spaces and counted the tokens for the whole document.

The tokenizers

We counted tokens with seven tokenizers that are publicly available:

TokenizerUsed byHow we ran it
o200k_baseOpenAI's GPT-4o and GPT-5 family (OpenAI's tiktoken library maps gpt-5 models to o200k_base)gpt-tokenizer npm package
cl100k_baseOpenAI's older GPT-4 and GPT-3.5 modelsgpt-tokenizer npm package
Gemma 3Google's open Gemma 3 models. An open-source port of Google's Python SDK states that Gemini models use the same tokenizer@lenml/tokenizer-gemma3 npm package
TekkenMistral modelsMistral's mistral-common Python package
DeepSeek V3DeepSeek V3 models@lenml/tokenizer-deepseek_v3 npm package
Llama 3Meta's Llama 3 modelsllama3-tokenizer-js npm package
Qwen 3Alibaba's Qwen 3 models@lenml/tokenizer-qwen3 npm package

Anthropic does not publish a tokenizer for current Claude models, so Claude is not in this test. You can measure Claude's token counts through its token-counting API, which we plan to add in a follow-up.

What the ratio means

For each tokenizer, we divided the token count of each language's document by the token count of the English document. A ratio of 1.65 for Hindi means the Hindi document needed 65% more tokens than the English one to carry the same text.

Our results match independent research

A 2026 paper, "The Tokenizer Tax", measured 14 Indian languages on a different corpus (FLORES-200). It found an average premium of 8.0x under the older cl100k tokenizer, and that multilingual tokenizers such as o200k cut the average tax by 73%.

Our numbers line up closely. We measured an average of 7.48x on cl100k and 1.99x on o200k across our nine languages. That is a reduction of about 73%, the same figure, from a different corpus and a different set of languages. The agreement gives us confidence that the pattern is real, not an artefact of the UDHR text.

What the token tax means in rupees

Tokens turn into rupees directly. Here is the cost of 1,000 chatbot conversations on GPT-5.5, where each conversation has the equivalent of 1,500 English input tokens and 500 English output tokens. We use OpenAI's published GPT-5.5 prices ($5 per million input tokens and $30 per million output tokens), converted at ₹95.04 per dollar (27 October 2026), and apply each language's o200k ratio. GST is excluded.

LanguageToken multiplier (o200k)Cost per 1,000 conversationsExtra cost vs EnglishUsable context vs English
English1.00₹2,138none100%
Hindi1.65₹3,521+₹1,38361%
Bengali1.66₹3,551+₹1,41360%
Urdu1.70₹3,626+₹1,48859%
Gujarati1.71₹3,662+₹1,52458%
Malayalam1.80₹3,844+₹1,70656%
Marathi1.90₹4,063+₹1,92553%
Kannada2.18₹4,658+₹2,52046%
Tamil2.51₹5,357+₹3,21940%
Punjabi2.80₹5,984+₹3,84636%

A Tamil-language chatbot on GPT-5.5 pays about 2.5x what an English one pays for the same conversations. On a tokenizer with a 7x to 10x tax, the gap would be several times wider.

The last column matters as much as the cost. If a model accepts 128,000 tokens of context, a Tamil document can only use the equivalent of about 51,000 English tokens of that window. Long-document tasks such as summarising contracts or answering questions over manuals hit the limit sooner in Indian languages.

Output tokens carry the same tax, and output is priced 6x higher than input on GPT-5.5. Answers written in Indian languages are therefore the most expensive part of the bill.

Results language by language

Each language behaves a little differently. Here is what the measurement shows for each, with the GPT-5 family (o200k) and Gemma 3 figures side by side, since those were the two most efficient tokenizers.

Hindi

Hindi needed 1.65x English on o200k and 1.37x on the Gemma 3 tokenizer. On Llama 3, it was 2.99x, noticeably better than Llama 3's results for most other Indian languages, which suggests Hindi got more attention in its vocabulary. On Qwen 3 and the older cl100k, Hindi cost more than 5x English.

Bengali

Bengali was the closest to English of any language in the test on the Gemma 3 tokenizer, at 1.14x. On o200k it was 1.66x, almost identical to Hindi. On Llama 3 and cl100k, it jumped to about 6x, so model choice matters a great deal for Bengali apps.

Urdu

Urdu was consistently one of the cheaper languages: 1.70x on o200k and 1.58x on Gemma 3. A likely reason is that its Perso-Arabic script overlaps with Arabic and Persian, which many multilingual tokenizers cover well.

Gujarati

Gujarati sat at 1.71x on o200k but rose to 1.82x on Gemma 3, one of the few languages where o200k won. On Llama 3 and cl100k it cost over 8x English.

Malayalam

Malayalam came in at 1.80x on o200k and 1.65x on Gemma 3. The Tokenizer Tax paper found Malayalam to be the most expensive language under cl100k at 13.0x. We measured 8.74x on our corpus, still among the highest, with the difference likely due to the different source text.

Marathi

Marathi uses Devanagari like Hindi but cost more: 1.90x on o200k and 1.45x on Gemma 3. Marathi attaches grammatical endings to words, so its words are longer. In our corpus, the Marathi text had 1,614 words against 2,128 in Hindi for the same content, and each longer word splits into more pieces.

Kannada

Kannada was expensive almost everywhere: 2.18x on o200k, 2.13x on Gemma 3 and 9.44x on Llama 3 and cl100k. Builders of Kannada products should test tokenizers carefully before committing to a model.

Tamil

Tamil carried the second-highest tax on o200k at 2.51x, though Gemma 3 cut that to 1.89x. Its long, agglutinative words averaged about 3.86 tokens each on o200k, more than three times the English rate.

Punjabi

Punjabi was the most expensive language on most tokenizers: 2.80x on o200k and 2.90x on Gemma 3. Part of this comes from the translation, which is about 30% longer in words than the English original, but even allowing for that, Gurmukhi text tokenizes poorly on most models.

Why Indian languages cost more tokens

Tokenizers split text into pieces they learned during training, using a method called byte-pair encoding. Frequent sequences become single tokens. Rare ones are broken into smaller pieces, sometimes down to individual bytes.

Three things push Indian languages into smaller pieces:

  1. Training data skew. Tokenizers learn from their training data, which has historically been dominated by English and a few other languages. Common English words become single tokens. Many Hindi or Tamil words do not.
  2. Script complexity. Indic scripts combine consonants, vowel signs and conjuncts. A single visible syllable can be several Unicode characters, and each Unicode character in these scripts takes three bytes in UTF-8.
  3. Failed merges fall back to bytes. When the tokenizer has no learned piece for a sequence, it falls back to raw bytes. The Tokenizer Tax paper found that this merge failure explains the tax very closely, with a correlation of 0.89.

You can see the effect in tokens per word. On o200k, English used about 1.13 tokens per word in our test, Hindi about 1.53 and Tamil about 3.86. Tamil words are long and agglutinative, and each one is split into many pieces.

Part of the gap also comes from the translations themselves. The Punjabi UDHR has 2,293 words against 1,747 in English, so some of its tax reflects a wordier translation rather than the tokenizer alone. That is realistic, because real Punjabi text is what you will send, but it means the language ratios mix tokenizer efficiency with translation style.

What about Hinglish?

Many Indian users do not write in Devanagari at all. They type Hindi in Roman letters, mixed with English words: "Meri API key kaam nahi kar rahi." We wanted to know whether that is cheaper.

The UDHR has no Hinglish version, so we wrote a small test set: 10 everyday support questions, each in English, Hindi in Devanagari and romanised Hinglish. This is a small, hand-written sample, not a benchmark, so treat the numbers as indicative.

TokenizerHindi (Devanagari) vs EnglishHinglish (romanised) vs English
GPT-5 family (o200k)1.581.48
Gemini (Gemma 3 tokenizer)1.261.30
Mistral (Tekken)1.901.54
DeepSeek V33.141.59
Llama 32.581.60
Qwen 34.541.60
GPT-4 (cl100k, older)5.021.60

The result depends on the tokenizer:

Romanised Hinglish settles at around 1.5x to 1.6x English on every tokenizer we tried, because its words are simply longer than their English equivalents. Do not force users to switch scripts to save money. Pick a model whose tokenizer handles the script your users already type.

How to reduce the token tax

You cannot change a model's tokenizer, but you can control how much Indian-language text you send and which model reads it.

  1. Choose the model with the tokenizer in mind. For Indian-language workloads, a model with a multilingual tokenizer can cost a third or less of an English-centred one for the same text. Test your own content, because the best tokenizer varies by language.
  2. Keep instructions in English. System prompts, tool definitions and formatting rules do not need to be in the user's language. Write them in English, and let only the user-facing content be in Hindi, Tamil or another language.
  3. Cache the fixed parts. Long system prompts and reference documents can be cached by most providers, which cuts their input cost by 10x to 50x. That helps every language, but it helps most where tokens are expensive. Our guide to what AI coding agents really cost explains how caching works and what breaks it.
  4. Raise output limits deliberately. If you set max_tokens for English answers, an answer in Tamil may be cut off at about 40% of the intended length. Scale output limits by the language's multiplier.
  5. Count real tokens, not words. Estimate budgets with each provider's usage numbers or tokenizer, not with word counts. A 1,000-word Tamil document is not the same cost as a 1,000-word English one.
  6. Consider translating long reference material once. For retrieval systems, storing and searching an English version of large reference documents can reduce cost, but it adds translation errors. Measure answer quality before switching.
  7. Watch context limits in long documents. Plan chunk sizes in tokens, and expect Indian-language chunks to hold less text than English chunks of the same token size.

Reproduce our measurement

You can rerun the core of this test in a few lines of JavaScript. This example counts tokens with the GPT-5 family tokenizer.

// npm install gpt-tokenizer
import { encode } from 'gpt-tokenizer/encoding/o200k_base';
import fs from 'node:fs';

const clean = (t) => t.replace(/\s+/g, ' ').trim();
const english = clean(fs.readFileSync('eng.txt', 'utf8'));
const hindi = clean(fs.readFileSync('hin.txt', 'utf8'));

const e = encode(english).length;
const h = encode(hindi).length;
console.log({ english: e, hindi: h, ratio: (h / e).toFixed(2) });

The UDHR text files come from the udhr2 corpus in the NLTK data repository on GitHub. Swap in other tokenizers from the table above to compare models.

To measure what you are actually billed, send the same text to the model through an API with a one-token output limit and read the input token count from the response's usage field. That captures any formatting tokens the provider adds.

What this means for Indian AI products

The token tax is a cost of building for Bharat. It is also a design choice you can partly control.

On Velona, you can call GPT, Gemini, DeepSeek, Qwen, Mistral and other models through one OpenAI-compatible API and pay in rupees with UPI, which makes it easy to compare real token counts on your own content before you choose. See how to pay for AI APIs with UPI and our guide to the best affordable AI API for Indian students and indie developers.

Frequently asked questions

Does AI cost more in Hindi than in English?

Yes. In our measurement, the same text needed 1.65x as many tokens in Hindi as in English on the GPT-5 family tokenizer, and 1.37x on the Gemma 3 tokenizer that Gemini tooling uses. On older tokenizers it was over 5x. Since AI APIs bill per token, Hindi costs more for the same content.

How many tokens is a Hindi word?

On the GPT-5 family's o200k tokenizer, Hindi averaged about 1.53 tokens per word in our UDHR test, against about 1.13 for English. Tamil averaged about 3.86 tokens per word because its words are longer.

Which AI model is cheapest for Indian languages?

It depends on the model's price and its tokenizer together. In our test, the Gemma 3 tokenizer used by Gemini tooling had the lowest average token tax across nine Indian languages (1.77x English), followed by the GPT-5 family's o200k (1.99x). Multiply the tax by each model's per-token price for your language to find the cheapest option.

Is Hinglish cheaper than Hindi for AI?

On older tokenizers, yes, often by more than half. On modern multilingual tokenizers, the saving is small, and on Gemini's tokenizer, Devanagari Hindi was slightly cheaper than romanised Hinglish in our small sample.

Why do Indian languages use more tokens?

Tokenizers are trained mostly on English-heavy data, Indic scripts use several Unicode characters per syllable, and sequences the tokenizer has not learned fall back to raw bytes. Research in 2026 found that this merge failure explains the tax closely.

Does the token tax affect the context window?

Yes. A Tamil document using a 2.51x multiplier fits only about 40% as much text into the same context window as English. Long-document tasks hit the limit sooner in Indian languages.

Can I check token counts before I send a request?

Yes. Use the provider's tokenizer library where one exists, such as OpenAI's tiktoken, or send a request with a one-token output limit and read the input token count from the API response.

Limits of this study

Sources