Build a Real-Time Side-by-Side Model Evaluator

When selecting a Large Language Model (LLM) for production, relying only on generic benchmark charts or marketing claims isn't enough. A model that scores high on academic tests might be too slow for real-time customer support or too expensive for batch document processing.

To decide with data, you need to run the exact same prompt across several providers at once and measure response latency, token usage, output quality, and exact cost in Indian Rupees.

In this tutorial, we will build a Side-by-Side Model Evaluator in Python using asyncio, httpx, and Velona's AI Gateway. Comparing four models on one prompt costs well under ₹1.

Architecture of Concurrent Model Evaluation

Traditional multi-provider benchmarking means separate SDKs, separate API keys, and separate foreign-currency billing accounts for OpenAI, Anthropic, and Google. With Velona, every provider is reachable with one key and one request format through the native endpoint (/gateway/v1/inference/run), or through the OpenAI-compatible endpoint at https://velona.in/v1.

You submit a test prompt.
An async Python script sends concurrent requests to several models, for example openai/gpt-4o-mini, anthropic/claude-haiku-4.5, google/gemini-2.5-flash and deepseek/deepseek-v4-flash.
The script measures round-trip latency and reads the token counts each call returns.
Results are printed side by side with the cost of each call in rupees.

Step 1: Set Up Your Velona Account and Key

Create an account on the Velona Registration Page.
Top up on the Prepaid Wallet Dashboard starting at ₹10 using UPI (GPay, PhonePe) or RuPay.
Go to API Key Management and create a key named model-evaluator-app.
Set a budget on the key to keep testing costs predictable.

Step 2: Build the Asynchronous Evaluator in Python

Install the async HTTP client (asyncio ships with Python, so do not pip install it):

pip install httpx

Create a file named evaluator.py:

import os
import time
import asyncio
import httpx

VELONA_API_KEY = os.environ["VELONA_API_KEY"]
VELONA_URL = "https://velona.in/gateway/v1/inference/run"
HEADERS = {
    "Authorization": f"Bearer {VELONA_API_KEY}",
    "Content-Type": "application/json",
}

# ₹ per 1M tokens (input, output). Copy current values from velona.in/pricing
PRICES = {
    "openai/gpt-4o-mini": (15.54, 62.17),
    "anthropic/claude-haiku-4.5": (103.61, 518.05),
    "google/gemini-2.5-flash": (31.08, 259.02),
    "deepseek/deepseek-v4-flash": (2.90, 5.80),
}


async def evaluate_model(client: httpx.AsyncClient, model: str, prompt: str):
    payload = {"model": model, "turns": [{"role": "user", "content": prompt}]}
    start = time.perf_counter()
    try:
        response = await client.post(VELONA_URL, json=payload, headers=HEADERS, timeout=60.0)
        elapsed_ms = (time.perf_counter() - start) * 1000
        response.raise_for_status()

        data = response.json()["data"]          # native endpoint: data.output, data.usage
        usage = data.get("usage", {})
        p_tok = usage.get("prompt_tokens", 0)
        c_tok = usage.get("completion_tokens", 0)
        in_price, out_price = PRICES[model]
        cost_inr = (p_tok * in_price + c_tok * out_price) / 1_000_000

        return {
            "model": model,
            "latency_ms": round(elapsed_ms),
            "prompt_tokens": p_tok,
            "completion_tokens": c_tok,
            "cost_inr": cost_inr,
            "output": data["output"].strip(),
        }
    except Exception as e:
        return {"model": model, "error": str(e)}


async def run_evaluation(prompt: str):
    async with httpx.AsyncClient() as client:
        tasks = [evaluate_model(client, m, prompt) for m in PRICES]
        return await asyncio.gather(*tasks)


if __name__ == "__main__":
    test_prompt = ("Write a Python function to check if a string is a valid "
                   "palindrome, with a docstring and edge cases.")
    print(f"--- Evaluating: '{test_prompt}' ---\n")

    for res in asyncio.run(run_evaluation(test_prompt)):
        print(f"=== {res['model']} ===")
        if "error" in res:
            print(f"Error: {res['error']}\n")
            continue
        print(f"Latency: {res['latency_ms']} ms")
        print(f"Tokens: prompt={res['prompt_tokens']}, completion={res['completion_tokens']}")
        print(f"Cost: ₹{res['cost_inr']:.4f}")
        print(f"Preview:\n{res['output'][:200]}...\n")

Your usage page shows the exact amount Velona charged for each call, so you can confirm the script's estimates.

What an Evaluation Run Costs

The palindrome prompt above is about 30 input tokens and 300 to 500 output tokens per model. At the time of writing that works out to roughly:

Model	Approx. cost per call
DeepSeek V4 Flash	₹0.002
GPT-4o Mini	₹0.025
Gemini 2.5 Flash	₹0.10
Claude Haiku 4.5	₹0.21

That is about ₹0.35 to compare all four, so ₹10 covers around 30 full comparison runs. The spread is the point: for many prompts the cheapest model's answer is good enough, and this script shows you exactly when it is.

Key Metrics for Model Comparison

Metric	Why It Matters	What to Look For
Response latency (ms)	Time to receive the full reply. Critical for interactive tools.	Fast enough for your use case; stream for chat UIs
Token efficiency	How many output tokens the model needs to answer correctly.	Concise, accurate answers without padding
₹ cost per call	Exact cost from input and output token counts.	Lowest cost that meets your quality bar
Format adherence	Whether the model follows structure (JSON, Markdown, code blocks).	Consistent output your code can parse

Using Velona's Built-In Evaluator and Leaderboards

Don't want to write a script? Velona has these tools built in:

Model Evaluator: run a prompt side by side across models in the browser with streaming and rupee cost per answer.
AI Benchmark Leaderboard: intelligence, coding, speed and cost data across top models.
Model Pricing: per-1M-token prices in rupees for every model.

For a ready-made comparison on coding tasks, see the cheapest AI models for coding agents in ₹.

Developer Tools and Debugging

When inspecting JSON responses or authorization headers, use Velona's free developer tools, such as the JSON Formatter and JWT Decoder. They run in your browser and never touch your wallet.

Conclusion

A multi-model evaluator helps you avoid vendor lock-in and pick a model on real performance and real cost. With Velona's single key and rupee billing, Indian developers can benchmark OpenAI, Claude, Gemini and DeepSeek side by side and pay with a UPI top-up.

Ready to benchmark your prompts? Create a free Velona account, add ₹10, and run your first comparison today.

Build a Real-Time Model Evaluator Playground (OpenAI vs Claude vs Gemini)