Build a Real-Time Model Evaluator Playground (OpenAI vs Claude vs Gemini)

When selecting a Large Language Model (LLM) for production, relying solely on generic benchmark charts or marketing claims isn't enough. A model that scores high on academic tests might be too slow for real-time customer support or too expensive for batch document processing.

To make data-driven decisions, developers need to test the exact same prompt across multiple model providers simultaneously, measuring latency (time-to-first-token), generation speed (tokens/sec), output accuracy, and exact cost in Indian Rupees (INR).

In this tutorial, we will build a Real-Time Side-by-Side Model Evaluator in Python using asyncio, httpx, and Velona's Unified AI Gateway.

Architecture of Concurrent Model Evaluation

Traditional multi-provider benchmarking requires managing distinct SDKs, separate API keys, and different payment accounts for OpenAI, Anthropic, and Google. With Velona, every provider is accessible through a single OpenAI-compatible REST endpoint (/gateway/v1/inference/run)[cite: 1].

1. The user submits a test prompt and system instruction.
2. An asynchronous Python script dispatches concurrent HTTP POST requests to multiple target models (e.g., openai/gpt-4o, anthropic/claude-3.5-sonnet, and google/gemini-flash-1.5)[cite: 1].
3. The script measures exact round-trip response latency (in milliseconds) and token counts for each provider.
4. Results are rendered side-by-side with total cost calculated in native INR paise[cite: 1].

Step 1: Set Up Your Velona Account & Key

To run concurrent queries across 300+ models under a single account[cite: 1]:

1. Create an account on the Velona Registration Page[cite: 1].
2. Top up your balance on the Prepaid Wallet Dashboard starting at ₹10 using UPI (GPay, PhonePe) or RuPay[cite: 1].
3. Navigate to API Key Management and generate an API key tagged for model-evaluator-app[cite: 1].
4. Set an optional monthly budget limit on the key to manage testing expenses safely[cite: 1].

Step 2: Build the Asynchronous Evaluator in Python

Install the required asynchronous HTTP client:

pip install httpx asyncio

Create a file named evaluator.py and add the following code:

import time
import asyncio
import httpx

VELONA_API_KEY = "YOUR_VELONA_API_KEY"
VELONA_URL = "https://velona.in/gateway/v1/inference/run"

HEADERS = {
    "Authorization": f"Bearer {VELONA_API_KEY}",
    "Content-Type": "application/json"
}

# Target Models to Compare Simultaneously
TARGET_MODELS = [
    "openai/gpt-4o",
    "anthropic/claude-3.5-sonnet",
    "google/gemini-flash-1.5"
]

async def evaluate_model(client: httpx.AsyncClient, model_name: str, prompt: str):
    payload = {
        "model": model_name,
        "turns": [
            {"role": "user", "content": prompt}
        ]
    }
    
    start_time = time.perf_counter()
    try:
        response = await client.post(VELONA_URL, json=payload, headers=HEADERS, timeout=30.0)
        elapsed_ms = (time.perf_counter() - start_time) * 1000
        
        data = response.json()
        content = data["choices"][0]["message"]["content"]
        usage = data.get("usage", {})
        
        return {
            "model": model_name,
            "latency_ms": round(elapsed_ms, 2),
            "prompt_tokens": usage.get("prompt_tokens", 0),
            "completion_tokens": usage.get("completion_tokens", 0),
            "output": content.strip()
        }
    except Exception as e:
        return {
            "model": model_name,
            "error": str(e)
        }

async def run_evaluation(prompt: str):
    async with httpx.AsyncClient() as client:
        tasks = [evaluate_model(client, model, prompt) for model in TARGET_MODELS]
        results = await asyncio.gather(*tasks)
        return results

# Example Execution
if __name__ == "__main__":
    test_prompt = "Write a Python function to check if a string is a valid palindrome, with docstring and edge cases."
    print(f"--- Running Concurrent Evaluation for Prompt: '{test_prompt}' ---\n")
    
    results = asyncio.run(run_evaluation(test_prompt))
    
    for res in results:
        print(f"=== Model: {res['model']} ===")
        if "error" in res:
            print(f"Error: {res['error']}\n")
            continue
        print(f"Latency: {res['latency_ms']} ms")
        print(f"Tokens Used: Prompt={res['prompt_tokens']}, Completion={res['completion_tokens']}")
        print(f"Response Preview:\n{res['output'][:200]}...\n")

Key Metrics for Model Comparison

When running evaluations across candidate models, track four key dimensions:

Evaluation Metric	Why It Matters	Ideal Target
Response Latency (ms)	Time required to receive the full output payload. Critical for interactive tools.	< 1,500 ms for fast models[cite: 1]
Token Efficiency	Number of completion tokens generated to answer the prompt correctly.	Concise, accurate outputs without verbosity
INR Cost per Call	Exact cost charged in paise based on input/output token counts[cite: 1].	Lowest cost meeting quality thresholds[cite: 1]
Format Adherence	Whether the model strictly follows structural instructions (JSON, Markdown, code blocks).	100% adherence without hallucinated syntax

Using Velona's Built-In Playground & Leaderboards

Don't want to write custom benchmark scripts? Velona provides built-in tools directly on the dashboard:

• Model Evaluator Playground: Run prompts side-by-side across multiple models in real time with live SSE streaming, time-to-first-token counters, and INR cost calculations[cite: 1].
• AI Benchmark Leaderboard: Browse live Artificial Analysis performance data covering Intelligence Index, Coding Accuracy, Output Speed (tokens/sec), and Blended Cost across top LLMs[cite: 1].
• Model Pricing Index: Sort 300+ models by per-1M token pricing calculated transparently in Indian Rupees[cite: 1].

Developer Tools & Debugging

When analyzing JSON responses, calculating execution overheads, or inspecting authorization headers, utilize Velona's 49 Life Time Free Developer Tools[cite: 1, 1]. Utilities like the JSON Formatter, Character Counter, and HTTP Header Inspector operate 100% free without consuming wallet funds[cite: 1]. You can also test quick single-model responses in the Free Public Chat Playground[cite: 1] or review detailed call logs on the Usage Dashboard[cite: 1].

Conclusion

Building a multi-model evaluator enables engineering teams to avoid vendor lock-in and select the optimal AI provider based on real performance data. With Velona's unified REST API gateway and native UPI billing, Indian developers can benchmark and deploy top-tier AI models effortlessly[cite: 1].

Ready to benchmark your prompts? Sign up for a free Velona account and start evaluating today[cite: 1]!