Build a WhatsApp Expense Tracker with Vision OCR

Tracking daily expenses manually is tedious. Whether it's paper restaurant bills, grocery receipts, or digital UPI payment screenshots, most people forget to log their transactions into spreadsheet apps or budgeting software.

In this tutorial, we will build an automated WhatsApp Expense Tracker Bot using Python, Twilio, and a multimodal vision model through Velona. Users snap a picture of a receipt or payment screenshot on WhatsApp, and the AI extracts the vendor name, total amount, date and category as structured JSON. Each receipt costs roughly ten paise to read.

Why Multimodal Vision LLMs Beat Traditional OCR

Legacy Optical Character Recognition (OCR) engines struggle with crumpled receipts, handwritten bills, low lighting, or complex table layouts. Multimodal models like Gemini 2.5 Flash and GPT-4o Mini combine visual text recognition with contextual understanding. They can tell that "CGST + SGST" is tax, distinguish sub-totals from the final amount, and categorise the spend ("Dining", "Groceries", "Fuel") automatically.

Prerequisites and Architecture

To follow along with this build, you will need:

A Velona Account with a funded wallet (starts at just ₹10 via UPI or RuPay).
A Twilio WhatsApp Sandbox number for receiving images.
Python 3.10+ with FastAPI.

pip install fastapi uvicorn twilio requests openai python-multipart

Step 1: Fund Your Wallet and Generate API Keys

Before handling image requests:

Log in to your Velona Wallet Dashboard and top up your balance with UPI. Usage is billed in rupees: the provider's price converted at the live rate, plus a 7% platform fee and a 0.5% forex buffer, all shown upfront on the pricing page.
Head to API Key Management and create a new key named expense-tracker-prod.
Set a budget cap on the key to protect against unexpected image volume spikes.

Step 2: Build the FastAPI Vision Webhook Backend

When a user sends an image via WhatsApp, Twilio forwards the media URL to our webhook. The backend downloads the image, encodes it to base64, and sends it to Velona's OpenAI-compatible endpoint at https://velona.in/v1, using the standard OpenAI image message format.

import os
import base64
import requests
from fastapi import FastAPI, Form, Response
from openai import OpenAI
from twilio.twiml.messaging_response import MessagingResponse

app = FastAPI()

client = OpenAI(
    base_url="https://velona.in/v1",
    api_key=os.environ["VELONA_API_KEY"],
)
VISION_MODEL = "google/gemini-2.5-flash"

TWILIO_SID = os.environ["TWILIO_ACCOUNT_SID"]
TWILIO_TOKEN = os.environ["TWILIO_AUTH_TOKEN"]

PROMPT = """Analyze this receipt or payment screenshot and extract the transaction.
Return ONLY a raw JSON object with these keys:
{
  "vendor": "Merchant name",
  "amount_inr": 0.00,
  "date": "YYYY-MM-DD",
  "category": "Dining / Groceries / Utilities / Travel / Other",
  "summary": "Brief 1-line item list"
}"""


def download_image(url: str):
    # Twilio media URLs require your account credentials by default
    res = requests.get(url, auth=(TWILIO_SID, TWILIO_TOKEN), timeout=30)
    res.raise_for_status()
    mime = res.headers.get("Content-Type", "image/jpeg")
    return base64.b64encode(res.content).decode("utf-8"), mime


@app.post("/whatsapp-expense")
async def process_receipt(MediaUrl0: str = Form(None), Body: str = Form("")):
    twiml = MessagingResponse()

    if not MediaUrl0:
        twiml.message("Please send a photo or screenshot of a receipt or invoice.")
        return Response(content=str(twiml), media_type="application/xml")

    b64_img, mime = download_image(MediaUrl0)

    completion = client.chat.completions.create(
        model=VISION_MODEL,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": PROMPT},
                {"type": "image_url",
                 "image_url": {"url": f"data:{mime};base64,{b64_img}"}},
            ],
        }],
        max_tokens=300,
    )
    result = completion.choices[0].message.content

    twiml.message(f"Receipt processed!\n\n{result}")
    return Response(content=str(twiml), media_type="application/xml")

From here you can parse the JSON with json.loads and append a row to Google Sheets or your database. New to the OpenAI SDK with Velona? See how to use an AI API with Python.

Step 3: Choose a Vision Model and Control Cost

Vision requests are billed for the image tokens as well as the text, so images cost more than plain chat. Approximate cost per receipt at the time of writing:

Model ID	₹ per 1M input / output	Approx. per receipt	Best for
`google/gemini-2.5-flash`	₹31.08 / ₹259.02	about ₹0.10	Default: phone photos, Hindi text on bills, screenshots
`openai/gpt-4o-mini`	₹15.54 / ₹62.17	₹0.05 to ₹0.25	Clear digital screenshots (cost depends on image size)

Two simple ways to keep costs down: resize photos to about 1,000 pixels on the long side before sending (more pixels means more image tokens), and cap max_tokens since the JSON reply is short. At about ₹0.10 per receipt, ₹10 covers roughly 100 receipts. Each request's exact rupee cost appears on your usage dashboard.

Developer Tools and Testing

When debugging base64 strings or validating the JSON output, use Velona's free developer tools. The Base64 Converter and JSON Formatter run in your browser and never touch your wallet.

To compare extraction quality across models before writing code, run the same receipt through the Model Evaluator.

Conclusion

A WhatsApp expense tracker built on a multimodal model removes manual data entry and turns receipt photos into structured finance logs in seconds. With Velona, Indian developers can build vision-powered tools and pay for them with a UPI top-up instead of an international card.

Ready to build your multimodal AI app? Create a free Velona account, add ₹10, and process your first hundred receipts.

Build a Smart WhatsApp Expense Tracker with Vision OCR using Multimodal AI