How to Evaluate AI Coding Models on Your Own Code: Cost per Solved Task in ₹ (2026 Guide)

The cheapest model per token is often not the cheapest model per finished task, and public leaderboards can no longer settle the question for you. In February 2026 OpenAI stopped reporting SWE-bench Verified after finding contamination and flawed tests. The same model can score differently on two identical runs. The reliable answer comes from a small private evaluation on your own code: 20 to 30 tasks with tests, a few runs each, and one number that matters for your budget, the rupee cost per solved task.

This guide shows how to set that up in an afternoon, what to log, how to compute the metrics, and how to read the results without fooling yourself. It works for comparing models, comparing agents, or testing whether routing cheap and strong models saves money without hurting quality.

Why public coding benchmarks are not enough in 2026

Leaderboards are still useful for a shortlist. They are a poor basis for spending decisions, for three reasons.

• Contamination. SWE-bench Verified uses public GitHub issues and fixes that models may have seen in training. A 2025 study, The SWE-Bench Illusion, found models could name the right file to edit far more often on SWE-bench repositories than on others, with accuracy up to 76% on SWE-bench but at most 53% elsewhere. OpenAI's own audit of problems its models failed found many with tests that reject correct fixes.
• Saturation. Top models now cluster within a few points of each other on Verified, so small differences say little. Harder sets such as SWE-bench Pro spread models out again, but they are still not your code.
• No cost column. A leaderboard tells you how often a model succeeds, not what each success costs in rupees with your agent, your prompts and your repository.

That is why our coding agents research overview warned against treating a benchmark score as proof a model will work on your codebase. A private eval fixes all three problems: the tasks are unseen, you choose the difficulty, and you see the bill.

The metrics that matter

Cost per solved task is the one to put at the top of your report. It folds in failed attempts, retries and long runs, which per token prices hide.

Why cheap per token is not always cheap per solve

Here is the arithmetic with illustrative numbers. These are not benchmark results. They show how the metric works.

Two things often change the picture on real code: a cheap model that loops for 100 steps on hard tasks, and a cheap model whose passing runs quietly break other tests. The regression rate and the cost distribution catch both.

Step by step: build your own coding model evaluation

Step 1. Pick 20 to 30 tasks with objective pass or fail checks

Use work from your own repositories, ideally from after the models' training cutoffs, so nobody has seen the answer. Every task needs a check a script can run: a failing test that should pass, a type check, or a command with an expected output.

Write each task as a short markdown file: the instruction exactly as the agent will see it, the starting commit, and the command that decides pass or fail. Mix easy and hard. If every model solves every task, the eval tells you nothing.

Step 2. Standardise everything except the thing you are testing

If you are testing context engineering instead of models, hold the model fixed and change only the context, for example with and without your AGENTS.md. Our context engineering research review lists seven practices worth testing this way.

Step 3. Repeat each run

Model output varies even at temperature 0. One study found accuracy swings of up to 15% across identical runs, and the tau-bench authors found GPT-4o succeeded on fewer than 25% of retail tasks across all 8 of 8 tries even though it passed about 61% on a single try. Three runs per task is a practical minimum. Five is better for close calls.

Step 4. Log the evidence for every API call

Every OpenAI compatible API returns token counts with each response. Log them per call, tagged with the task and run, and keep measured values apart from estimates. If you use an agent that does not expose usage, the per request rupee cost in your Velona usage dashboard is the measured source.

import csv, os, time
from openai import OpenAI

client = OpenAI(api_key=os.environ["VELONA_API_KEY"], base_url="https://velona.in/v1")

def logged_call(task_id, run_id, model, messages, log="calls.csv"):
    t0 = time.time()
    resp = client.chat.completions.create(model=model, messages=messages, max_tokens=2000)
    u = resp.usage
    with open(log, "a", newline="") as f:
        csv.writer(f).writerow([task_id, run_id, model, u.prompt_tokens,
                                u.completion_tokens, round(time.time() - t0, 2)])
    return resp

Step 5. Compute the metrics

import pandas as pd

runs = pd.read_csv("runs.csv")
summary = runs.groupby("model").agg(
    success_rate=("passed", "mean"),
    median_cost=("total_inr", "median"),
    total_cost=("total_inr", "sum"),
    solved=("passed", "sum"),
)
summary["cost_per_solve"] = summary.total_cost / summary.solved

per_task = runs.groupby(["model", "task"]).passed.all()
summary["pass_all_runs"] = per_task.groupby("model").mean()

passed = runs[runs.passed == 1]
summary["regression_rate"] = passed.groupby("model").regressions.apply(lambda s: (s > 0).mean())
summary["median_time_to_pass"] = passed.groupby("model").wall_time.median()
print(summary.round(2))

Step 6. Read the results honestly

• Sample size. With 24 tasks, one task is about 4 percentage points. Treat gaps smaller than two or three tasks as a tie.
• Look at failures, not just totals. Read a few failed runs per model. A model that fails because of a flaky test is different from one that edits the wrong file.
• Check the cost spread. A low median with a few ₹200 runs means the model sometimes loops. A step limit or a fallback rule may fix it.
• Separate categories. A budget model may be fine for test writing and weak on refactors. That is a routing rule, not a reason to reject it.
• Rerun when prices or models change. Prices and model versions move every few months. Keep the task set and rerun it.

Use the same eval to test routing and trust

Once the task set exists, two more questions cost almost nothing to answer.

For a starting point on routing code, see our tutorial on building an AI model router in Python.

Frequently asked questions

Sources and method

This guide describes a method. It does not report benchmark results from Velona, and the worked example uses hypothetical success rates and step counts to illustrate the calculation. Per step rupee costs use Velona list prices on 16 October 2026 for 30,000 input and 1,000 output tokens.

• OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities (February 2026)
• The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason (ICSE 2026)
• Yao et al., tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (ICLR 2025)
• Atil et al., Non-Determinism of Deterministic LLM Settings
• Velona: Claude Sonnet 5 pricing in rupees
• Velona: DeepSeek price in India