Try Chat Free Docs Pricing AI Analysis Repo Insights Free Tools About Blog
Sign in Get started free
< Back to blog
Research Velona Team ·29 October 2026 ·16 min read

Is Thinking Mode Worth It? Reasoning Effort, Overthinking and Whether AI Can Predict How Hard a Question Is

Short answer: thinking mode is worth it for hard, multi-step problems that the model can already solve some of the time, such as tricky debugging, maths, planning and logic. It is usually wasted on easy questions, lookups, formatting and the routine steps of an agent loop, where it can multiply the cost of an answer by 8x or more for no gain in quality.

The best setup is not "always on" or "always off". It is to match the reasoning effort to the difficulty of each question. Research from 2024 to 2026 shows this can cut cost sharply without losing accuracy, and that models can estimate a question's difficulty before they start answering, though not perfectly.

This guide explains what thinking mode does, what it costs in rupees, what the research says about overthinking and difficulty prediction, and a practical recipe you can apply today.

Reasoning effort settings at a glance

Every major provider now lets you control how much a model thinks. The names differ, but the idea is the same.

Provider and modelSettingLevelsDefaultHow thinking is billed
OpenAI GPT-5.5reasoning_effortnone, low, medium, high, xhighmediumReasoning tokens count as output
Anthropic Claude (adaptive thinking)output_config.effortlow, medium, high, xhigh, max (varies by model)high on most models, medium on Opus 5.5 and Haiku 5.5Full thinking billed as output, even when only a summary is shown
DeepSeek V4 modelsthinking switch plus reasoning_effortThinking on or off, effort as documentedCheck the model's docsCheck DeepSeek's pricing page

Claude's adaptive thinking is worth understanding. Claude decides on each request whether to think and how much, and the effort setting is soft guidance rather than a fixed token budget. At low effort, Claude skips thinking on simple tasks.

How the controls behave in practice

The settings look alike, but they behave differently in ways that affect your bill.

Through an OpenAI-compatible gateway such as Velona, you pass reasoning_effort on the /v1/chat/completions endpoint, together with max_completion_tokens.

What thinking mode actually does

Thinking mode spends extra computation at answer time. The model writes out intermediate reasoning, checks its steps and sometimes explores alternatives before giving the final answer. Researchers call this test-time compute.

The landmark study is "Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters" by Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar (2024, presented at ICLR 2025). Two findings matter for anyone paying per token.

That second point is the most useful rule in this guide. Thinking helps a model finish what it can almost do. It does not give a model abilities it lacks.

What thinking mode costs, in rupees

Thinking tokens are billed as output tokens, and output is the most expensive kind of token. On every major model, output costs 3x to 6x more than input.

Here is the cost of one question with 2,000 input tokens and a 500-token visible answer, at four levels of hidden thinking. Prices are each provider's published US dollar rates, converted at ₹95.04 per dollar (27 October 2026), excluding GST. The thinking token counts are illustrative, not measured on one benchmark.

ModelNo thinking2,000 thinking tokens8,000 thinking tokens30,000 thinking tokens
DeepSeek V4.1 Flash (peak)₹0.11₹0.34₹1.03₹3.54
DeepSeek V4 Pro (peak)₹0.44₹1.19₹3.45₹11.73
Claude Sonnet 5.5₹0.86₹2.76₹8.46₹29.37
Claude Opus 5.5₹1.71₹5.51₹16.92₹58.73
GPT-5.5₹2.38₹8.08₹25.19₹87.91

At 8,000 thinking tokens, one answer costs about 8x to 11x more than with no thinking. At 30,000, it costs 27x to 37x more. For an app answering 1,000 questions a day on Claude Sonnet 5.5, that is the difference between about ₹855 and ₹8,459 a day at 8,000 thinking tokens.

Are 30,000 thinking tokens realistic? Yes, on hard questions. Tian Pan's May 2026 analysis gives an example of a complex query on Claude Opus using 20,000 to 40,000 thinking tokens for about 500 visible tokens. The same analysis notes that most observability dashboards miss reasoning tokens, so teams often discover them on the invoice.

Two billing details catch people out on Claude:

Check the real count in each response. On Claude, usage.output_tokens_details.thinking_tokens shows the thinking share of output.

Worked example: one app, three reasoning policies

Numbers per question are easy to ignore. Numbers per month are not. Take a hypothetical support and coding assistant that answers 1,000 questions a day on Claude Sonnet 5.5, with 2,000 input tokens and a 500-token answer each.

Assume, for illustration, that 70% of questions are easy, 20% are medium and 10% are genuinely hard. This split is not measured. Replace it with your own data.

PolicyWhat it doesCost per dayCost per 30 days
Always highEvery question gets about 30,000 thinking tokens₹29,367about ₹8.8 lakh
Always mediumEvery question gets about 8,000 thinking tokens₹8,459about ₹2.54 lakh
Difficulty-awareEasy: no thinking. Medium: 2,000 tokens. Hard: 30,000 tokens₹4,087about ₹1.23 lakh

The difficulty-aware policy costs about half of always-medium and about one seventh of always-high, while still giving the hard 10% the full 30,000-token budget. The question is whether you can sort questions well enough. A cheap difficulty check on a small model such as Claude Haiku 5.5 costs about ₹0.0012 per question at 120 input tokens, or about ₹1.24 per 1,000 questions, so the sorting itself is almost free. Getting it right is the hard part, which is what the research below is about.

Misrouting has a real cost too. Send a hard question to the no-thinking tier and you may get a wrong answer, a retry or an unhappy user. That is why the recipe later in this guide escalates on failure instead of trusting the first guess.

The overthinking problem

Reasoning models often think far too much about easy questions. The best-known study of this is "Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs" by Xingyu Chen and colleagues, presented at ICML 2025.

The authors describe overthinking as spending excessive computation on simple problems with minimal benefit. They proposed efficiency metrics that measure how much of a model's reasoning actually contributes to the answer, and a self-training method that shortened reasoning while preserving accuracy on GSM8K, MATH500, GPQA and AIME.

Later work found a telling pattern. The DiffAdapt paper (2025) reported a U-shaped entropy pattern in reasoning traces: entropy was high on easy problems, low on medium ones and high again on hard ones. In plain terms, models wander on easy questions as well as on hard ones.

Overthinking matters for cost and for speed. Every unnecessary thinking token adds to the bill and to the wait before the answer appears. Our explainer on the memory wall shows why generated tokens are the slow and expensive part of inference.

Can an AI predict how hard a question is before answering?

Yes, partly. This is one of the most active research questions of 2025 to 2026, and the answer depends on what the predictor can see.

Inside the model: difficulty is written in its activations

William Lugoloobi and Chris Russell at Oxford showed in "LLMs Encode How Difficult Problems Are" (October 2025) that a model's internal state encodes difficulty before it writes anything. Using simple linear probes across 60 models, they found that human-labelled difficulty could be read from the model's activations with a correlation of about 0.88 on AMC maths problems, and the signal improved with model size.

There was a twist. Difficulty measured by how often models actually fail was much harder to read and scaled poorly. Models seem to know how hard a problem looks to humans better than they know whether they themselves will fail.

A follow-up by the same group with UCL, "LLMs Encode Their Failures" (2026), turned this into a router. Probes read before generation predicted success with AUROC above 0.7 in most settings. A router that used those predictions to pick among five models:

The catch is access. Probes need the model's internal activations, which no API customer can see. They also weakened as reasoning budgets grew. On GPT-OSS-20B, the probe's AUROC fell from 0.78 at low reasoning to 0.64 at high.

DiffAdapt used a similar probe to pick one of three inference strategies (easy, normal or hard) per question. Against a fixed normal strategy, it cut tokens by 22.4% on Qwen3-4B and about 10% on two 7B reasoning models.

Outside the model: text-only predictors

If you call a model through an API, you only see text. That is the harder setting, and the research is thinner.

The honest caveat: many routers do not beat one good model

Routing sounds like free money. A large January 2026 benchmark says otherwise.

LLMRouterBench evaluated routing methods on over 400,000 instances from 21 datasets and 33 models. Many methods performed about the same under fair evaluation. Several recent approaches did not reliably beat simply using the best single model. The commercial OpenRouter router showed a negative gain of 24.7% against the best single model in that test.

The best routers still helped. They reached up to a 4% accuracy gain over the best single model, or up to 31.7% lower cost at the same accuracy. But all routers struggled on the hardest queries, where only one or two models could answer correctly.

The lesson: routing works best as a simple, well-tested rule, not as a magic box. Our own benchmark of cheap LLMs for production agents and our 2026 research review cover more of this work.

When thinking mode is worth it

Pulling the research together gives a practical guide by task type.

TaskRecommended effortWhy
Factual lookups, short answers, rewordingnone or lowNo multi-step reasoning to improve. Thinking adds cost and delay
Formatting, extraction, classificationnone or lowOverthinking risk is high, gains are small
Routine agent steps (reading files, applying small edits)lowAnthropic notes follow-up steps that only process tool results can skip thinking
Writing and summarisinglow or mediumPlanning helps long pieces a little, rarely enough to justify high effort
Debugging a tricky bug, multi-file refactorsmedium or highMulti-step reasoning with checkable steps is where thinking pays
Maths, logic puzzles, algorithm designhighThe classic case where test-time compute improves accuracy
Problems the model almost never solvesUse a stronger modelSnell et al. found extra thinking only helps where the model has some success already

Two more rules from the research and the provider docs:

A practical recipe: difficulty-aware reasoning effort

You do not need a research lab to get most of the benefit. Here is a five-step recipe that works through any OpenAI-compatible API.

Step 1: set a low default

Start every request at low or medium effort. Most real traffic is easier than people assume.

Step 2: escalate on signals you already have

Raise the effort when the request itself tells you it is hard:

Step 3: add a cheap difficulty check for ambiguous cases

When you cannot tell, ask a small, cheap model to rate the difficulty from 1 to 5 before you choose the effort. This mirrors the text-only approach in VDAR-Router in a simpler form. Treat it as a pattern to test on your own traffic, not a proven result.

from openai import OpenAI

client = OpenAI(base_url="https://velona.in/v1", api_key="YOUR_VELONA_KEY")

RATER = "<cheap-model-id>"      # a small, fast model
SOLVER = "openai/gpt-5-nano"    # any model that supports reasoning_effort

def rate_difficulty(question: str) -> int:
    r = client.chat.completions.create(
        model=RATER,
        messages=[
            {"role": "system", "content": "Rate how hard this request is for a strong AI model, from 1 (trivial) to 5 (very hard). Reply with one digit only."},
            {"role": "user", "content": question},
        ],
        max_tokens=2,
    )
    text = r.choices[0].message.content.strip()
    return int(text[0]) if text[:1].isdigit() else 3

def answer(question: str) -> str:
    level = rate_difficulty(question)
    effort = "low" if level <= 2 else "medium" if level == 3 else "high"
    r = client.chat.completions.create(
        model=SOLVER,
        reasoning_effort=effort,
        messages=[{"role": "user", "content": question}],
        max_completion_tokens=8000,
    )
    return r.choices[0].message.content

The rating call costs a fraction of a paisa on a small model. It pays for itself if it moves even a modest share of requests from high to low effort. On Velona, use stream=True for anything that may take longer than a minute, because non-streaming calls must finish within 90 seconds.

Step 4: for code, try cheap first and check with tests

Code has a free verifier: the test suite. Run the cheap setting first, run the tests and escalate only on failure. This simple cascade is often hard for a pre-generation predictor to beat, because it uses the real outcome instead of a guess.

Step 5: log reasoning tokens per request

Track thinking tokens alongside input and output. Alert on drift per request, not just on the monthly total. That is how you catch a prompt change that silently doubled the bill.

What this means for Indian developers and startups

Reasoning costs are paid in dollars and felt in rupees. For the full picture of agent costs, see What Does an AI Coding Agent Really Cost in India? At ₹95 to the dollar, the gap between a low-effort and a high-effort answer on a frontier model is easily ₹5 to ₹50 per question.

Three practical consequences follow.

  1. Default to a cheap reasoning model for most traffic. DeepSeek V4.1 Flash with moderate thinking (about 2,000 tokens) costs less per answer than Claude Sonnet 5.5 or GPT-5.5 with no thinking at all. Keep frontier models with high effort for the questions that need them.
  2. Prefer per-request control over per-app settings. One global high-effort setting is the most common cause of runaway reasoning bills.
  3. Budget per key. Give each app or feature its own API key with a monthly cap, so a spike in hard questions cannot drain your balance.

Frequently asked questions

Is thinking mode worth the extra cost?

It is worth it for hard, multi-step problems the model can already partly solve, such as debugging, maths and planning. For simple questions, formatting and routine agent steps, it usually adds cost and delay without improving the answer.

Should I use high or low reasoning effort?

Use low as the default and raise it per request when the task is clearly multi-step or a previous attempt failed. Keep the effort constant within a conversation, because changing it can break prompt caching.

Why do AI models overthink easy questions?

Reasoning models are trained to think at length, and they do not always judge when a problem is simple. Research presented at ICML 2025 documented this overthinking on simple maths and proposed methods to shorten reasoning without losing accuracy.

Can an AI tell how hard a question is before answering it?

Partly. Research shows a model's internal activations encode problem difficulty before it writes anything, and probes built on them can route questions to save 37% to 70% of cost at matched accuracy in published tests. Text-only predictors, which work through an API, are less studied but show promising results.

Do reasoning tokens cost the same as normal output tokens?

They are billed as output tokens at the model's output price. On Claude, you pay for the full thinking even when only a summary is shown, and earlier thinking left in the conversation is billed again as input.

What is the difference between thinking mode and reasoning effort?

Thinking mode is whether the model reasons before answering. Reasoning effort controls how much it reasons. Some providers expose both, such as DeepSeek's thinking switch plus reasoning_effort, while Claude's adaptive thinking uses a single effort setting.

Does LLM routing actually save money?

It can, but not automatically. A 2026 benchmark found the best routers cut cost by up to 31.7% at the same accuracy, while several methods, including one commercial router, did worse than using the best single model.

Limits of this guide

The rupee figures use published prices and illustrative thinking-token counts, not measurements from one benchmark. The research results cited come mostly from maths and reasoning benchmarks, and may not transfer directly to your traffic. We are running our own test of text-only difficulty prediction on coding tasks, measured in rupees per solved task, and will publish the results.

Sources