Build an AI Model Router in Python: Automatically Choose the Cheapest LLM

Most AI applications start with a simple setup. A user sends a prompt, the application sends it to a language model, and the model returns a response.

That works well when you are building a small application. It becomes less efficient when the application starts handling hundreds or thousands of requests.

The reason is simple. Not every request needs the same amount of reasoning, context, or model capability.

A request to classify a short message does not necessarily need the same model as a difficult coding problem. Sending both requests to an expensive model can increase your API bill without giving you a meaningful improvement in every response.

In this tutorial, we will build an AI model router in Python that automatically selects an appropriate LLM based on the type and complexity of the request.

We will use Velona's unified AI gateway so that the application can switch between different models while keeping the same API structure.

What Is an AI Model Router?

An AI model router is a small layer between your application and your language models. Instead of sending every request to the same model, the router decides which model should handle each request.

A basic application looks like this:

User
  ↓
Application
  ↓
One LLM
  ↓
Response

With a model router, the architecture becomes:

User
  ↓
Application
  ↓
AI Model Router
  ↓
Choose model
  ↓
LLM
  ↓
Response

The user does not need to know which model processed the request. The routing happens inside the application.

Why Use Multiple LLMs?

Different language models have different strengths, context windows, response speeds, and prices.

For example, a lightweight model can handle a large number of simple requests at a very low cost, while a more capable model can be reserved for tasks that need additional reasoning or context.

At the time of writing, Velona lists DeepSeek V4 Flash at ₹8.32 per million input tokens and ₹16.63 per million output tokens. Kimi K2.6 is listed at ₹97.66 per million input tokens and ₹411.18 per million output tokens, while Grok 4.20 is listed at ₹128.48 per million input tokens and ₹256.96 per million output tokens.

These prices are live rates and can change, so always check the Velona Pricing Index before making production routing decisions.

The Model Routing Strategy

For our first version, we will keep the routing logic simple.

The router will classify requests into three broad categories:

• Simple tasks: Short questions, classification, extraction, and other lightweight requests.
• Coding tasks: Programming questions, debugging, API implementation, and code generation.
• Complex tasks: Long prompts, detailed analysis, research-style questions, and requests that require more context.

We can then map these categories to different models.

Task type	Example model	Reason
Simple tasks	DeepSeek V4 Flash	Very low token cost
Coding tasks	Kimi K2.6	Strong coding and agentic capabilities
Complex tasks	Grok 4.20	Large context window and higher-cost capability tier

This is only one possible routing strategy. In a production application, you should benchmark the models against your own workload before deciding which model belongs in each category.

Setting Up the Python Project

We will use Python with httpx to communicate with the Velona API.

pip install httpx python-dotenv

Create a .env file and add your Velona API key:

VELONA_API_KEY=YOUR_API_KEY

Do not place your API key directly inside source code that will be committed to GitHub or another public repository.

Connecting to the Velona AI Gateway

Velona provides a single gateway endpoint for inference across its available models.

The native inference endpoint is:

https://velona.in/gateway/v1/inference/run

The model is selected using the model field in the request body.

This means that the rest of our Python application can remain unchanged when the router switches from one model to another.

Creating the Model Configuration

We can keep the model IDs in a dictionary so they are easy to change later.

import os
import httpx
from dotenv import load_dotenv

load_dotenv()

API_KEY = os.environ["VELONA_API_KEY"]
API_URL = "https://velona.in/gateway/v1/inference/run"

MODELS = {
    "cheap": "deepseek/deepseek-v4-flash",
    "coding": "moonshotai/kimi-k2.6",
    "complex": "x-ai/grok-4.20"
}

The advantage of keeping model IDs separate from the routing logic is that you can change the models without rewriting the application.

Building the Model Router

We can start with a lightweight rule-based router. This approach does not require an additional LLM call just to decide which model should handle the request.

def choose_model(prompt):
    text = prompt.lower()

    coding_words = [
        "python",
        "javascript",
        "typescript",
        "code",
        "debug",
        "function",
        "api",
        "program",
        "error",
        "bug"
    ]

    complex_words = [
        "analyse",
        "analyze",
        "compare",
        "architecture",
        "strategy",
        "research",
        "reason",
        "step by step",
        "in detail"
    ]

    if any(word in text for word in coding_words):
        return MODELS["coding"]

    if len(prompt) > 2500 or any(word in text for word in complex_words):
        return MODELS["complex"]

    return MODELS["cheap"]

This is intentionally simple. The goal is not to create a perfect classifier. The goal is to demonstrate how an application can make a model selection decision before sending the request.

Sending the Request to the Selected Model

Now we can create a function that sends the selected model and the user's prompt to Velona.

def ask_model(model, prompt):
    response = httpx.post(
        API_URL,
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        },
        json={
            "model": model,
            "turns": [
                {
                    "role": "user",
                    "content": prompt
                }
            ]
        },
        timeout=60
    )

    response.raise_for_status()

    return response.json()

Putting the Router Together

We can now connect the model selection function to the API call.

def run_router(prompt):
    model = choose_model(prompt)

    print(f"Selected model: {model}")

    result = ask_model(model, prompt)

    print(result["data"]["output"])

    return result


if __name__ == "__main__":
    prompt = input("Ask something: ")
    run_router(prompt)

Now try a few different requests.

Ask something: What is the capital of France?

Selected model: deepseek/deepseek-v4-flash

A programming request should be routed differently:

Ask something: Write a Python function that removes duplicate values from a list.

Selected model: moonshotai/kimi-k2.6

A long analysis request can be routed to the higher-cost tier:

Ask something: Analyse the architecture of a multi tenant SaaS platform in detail.

Selected model: x-ai/grok-4.20

The user does not have to manually select a model for any of these requests.

Tracking Token Usage and Cost

Routing only becomes useful when we can measure its effect.

Velona returns token usage information with inference responses. The response includes the number of prompt tokens, completion tokens, and total tokens used by the request.

We can capture those values in our application:

def get_usage(result):
    usage = result.get("data", {}).get("usage", {})

    return {
        "input_tokens": usage.get("prompt_tokens", 0),
        "output_tokens": usage.get("completion_tokens", 0),
        "total_tokens": usage.get("total_tokens", 0)
    }

You can store this information alongside the selected model and request type. After collecting enough requests, you can compare how much your application would have spent using one model for everything versus using the router.

Why Cost Tracking Matters

Suppose an application receives thousands of requests every month.

If most of those requests are short and simple, sending every request to a higher-cost model can result in unnecessary spending.

A router allows the application to reserve more expensive models for requests where their additional capabilities are actually useful.

For example, the current Velona pricing for DeepSeek V4 Flash is significantly lower than the listed prices for Kimi K2.6 and Grok 4.20.

However, price alone should not determine the routing decision.

A model that costs more but solves a difficult task correctly on the first attempt can sometimes be more economical than a cheaper model that requires several retries.

Using an LLM as the Router

The rule-based approach works for simple applications, but keywords are not enough to understand every request.

A more advanced router can use a low-cost model to classify the request before selecting the final model.

User prompt
     ↓
Cheap classifier
     ↓
Task type and complexity
     ↓
Select model
     ↓
Final LLM
     ↓
Response

For example, the classifier could return structured information such as:

{
  "task": "coding",
  "complexity": "medium"
}

The router could then map that result to a suitable model.

There is an important trade-off here. The classifier itself consumes tokens and adds another network request. If the classifier costs too much or adds too much latency, the routing layer can reduce the benefits it was designed to provide.

For this reason, a rule-based router is often a good starting point. You can introduce an LLM classifier after you have enough application data to justify the additional complexity.

Adding a Fallback Model

Production applications also need to handle model failures.

A model can temporarily become unavailable, return an error, or hit a rate limit. A fallback model allows the application to continue processing the request.

def run_with_fallback(prompt):
    primary = choose_model(prompt)

    try:
        return ask_model(primary, prompt)

    except httpx.HTTPError:
        fallback = MODELS["cheap"]
        return ask_model(fallback, prompt)

For a production system, handle specific HTTP status codes and add appropriate retry limits rather than treating every exception in exactly the same way.

Improving the Router Over Time

The simple router in this tutorial can be extended in several ways.

• Quality thresholds: Only route a task to a cheaper model if it meets your required quality level.
• Latency routing: Prefer faster models when response time is more important than maximum capability.
• Context-aware routing: Select models based on the size of the input and the required context window.
• Historical performance: Track which models perform best for each type of request.
• Budget limits: Prevent individual users or workflows from exceeding a predefined AI budget.
• Fallback chains: Try another model when the primary model fails or does not meet a quality threshold.

When Should You Use a Model Router?

A model router is most useful when your application handles different types of AI workloads.

• AI customer support systems
• Coding assistants
• Research applications
• Document processing pipelines
• Content generation systems
• Internal company assistants
• Multi-step AI agents

If your application performs one narrow task and a single model already provides the required quality at an acceptable price, a router may add unnecessary complexity.

The value becomes clearer when your application has multiple workloads and the difference in model pricing is significant.

Conclusion

AI applications do not always need the same model for every request.

A simple classification request can often use a low-cost model, while a difficult coding or reasoning task may justify a more capable model.

An AI model router gives your application a way to make that decision automatically.

In this tutorial, we built a basic Python router that selects between DeepSeek V4 Flash, Kimi K2.6, and Grok 4.20 using a single Velona API endpoint.

The next step is to benchmark the router against real application traffic. Measure token usage, response quality, latency, and total INR cost before deciding which routing strategy works best for your workload.

You can explore the current Velona model pricing, compare available models, and use the Velona API documentation to start building your own model routing system.