Build a Smart WhatsApp Expense Tracker with Vision OCR using Multimodal AI

Tracking daily expenses manually is tedious. Whether it's paper restaurant bills, grocery receipts, or digital UPI payment screenshots, most people forget to log their transactions into spreadsheet apps or budgeting software.

In this tutorial, we will build an automated WhatsApp Expense Tracker Bot using Python, Twilio, and Velona's Multimodal Vision Endpoints. Users simply snap a picture of a receipt or payment screenshot on WhatsApp, and the AI automatically parses the vendor name, total amount, tax breakdown, and date into structured JSON[cite: 1].

Why Multimodal Vision LLMs Beat Traditional OCR

Legacy Optical Character Recognition (OCR) engines struggle with crumpled receipts, handwritten bills, low lighting, or complex table layouts. Multimodal Large Language Models like GPT-4o, Claude 3.5 Sonnet, and Gemini Flash 1.5 combine raw visual text recognition with contextual understanding[cite: 1]. They can identify that "CGST + SGST" equals tax, distinguish sub-totals from total amounts, and categorize the spending (e.g., "Dining", "Groceries", "Fuel") automatically.

Prerequisites & Architecture

To follow along with this build, you will need:

• A Velona Account with a funded wallet (starts at just ₹10 via UPI/RuPay)[cite: 1].
• A Twilio WhatsApp Sandbox number for receiving images.
• Python 3.10+ with FastAPI and httpx or requests installed.

Step 1: Fund Your Wallet & Generate API Keys

Before handling image processing requests:

1. Log in to your Velona Wallet Dashboard and top up your account balance[cite: 1]. Velona charges in native Indian Rupees (INR) without foreign exchange conversion markups[cite: 1].
2. Head to API Key Management and create a new key tagged for expense-tracker-prod[cite: 1].
3. Set an optional budget cap in paise on your key to protect against unexpected image volume spikes[cite: 1].

Step 2: Build the FastAPI Vision Webhook Backend

When a user sends an image via WhatsApp, Twilio forwards the media URL to our webhook. The backend fetches the image, encodes it to base64, and dispatches it to Velona's single OpenAI-compatible endpoint (/gateway/v1/inference/run)[cite: 1].

import base64
import requests
from fastapi import FastAPI, Form, Response
from twilio.twiml.messaging_response import MessagingResponse

app = FastAPI()

VELONA_API_KEY = "YOUR_VELONA_API_KEY"
VELONA_URL = "https://velona.in/gateway/v1/inference/run"

def encode_image_from_url(image_url: str) -> str:
    # Fetch image bytes from Twilio media server (using HTTP basic auth if required)
    res = requests.get(image_url)
    return base64.b64encode(res.content).decode('utf-8')

@app.post("/whatsapp-expense")
async def process_receipt(MediaUrl0: str = Form(None), Body: str = Form("")):
    twiml = MessagingResponse()

    if not MediaUrl0:
        twiml.message("Please send a photo or screenshot of a receipt/invoice.")
        return Response(content=str(twiml), media_type="application/xml")

    # Encode image to Base64
    base64_img = encode_image_from_url(MediaUrl0)

    # Construct Vision Prompt Payload
    headers = {
        "Authorization": f"Bearer {VELONA_API_KEY}",
        "Content-Type": "application/json"
    }

    prompt_text = """Analyze this receipt/invoice image and extract transaction details.
Return ONLY a raw JSON object with these keys:
{
  "vendor": "Merchant name",
  "amount_inr": 0.00,
  "date": "YYYY-MM-DD",
  "category": "Dining / Groceries / Utilities / Travel / Other",
  "summary": "Brief 1-line item list"
}"""

    payload = {
        "model": "openai/gpt-4o",
        "turns": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt_text},
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"}
                    }
                ]
            }
        ]
    }

    # Call Velona AI Gateway
    res = requests.post(VELONA_URL, json=payload, headers=headers)
    result = res.json()["choices"][0]["message"]["content"]

    # Format reply back to WhatsApp
    reply_msg = f"Receipt Processed Successfully!\n\n{result}"
    twiml.message(reply_msg)

    return Response(content=str(twiml), media_type="application/xml")

Step 3: Cost Optimization for Vision OCR

Vision requests process both prompt text and high-resolution image tile tokens. To optimize cost without sacrificing visual accuracy[cite: 1]:

• High Accuracy: Use openai/gpt-4o or anthropic/claude-3.5-sonnet for dense, low-light receipts or handwritten restaurant bills[cite: 1].
• Budget-Friendly: Switch to google/gemini-flash-1.5 for clear, high-contrast digital UPI payment screenshots at a fraction of a paise per request[cite: 1].

You can verify current per-token rates for all 300+ models on the Velona Model Pricing Calculator[cite: 1] and monitor real-time wallet deductions on the Usage Logs Dashboard[cite: 1].

Developer Tools & Testing Utilities

When debugging base64 payload strings or testing JSON schema validation, leverage Velona's 49 Life Time Free Developer Tools[cite: 1, 1]. Tools like the Base64 Converter, JSON Formatter, and Image Utilities run completely free without requiring wallet credits[cite: 1].

To test different system prompt variations or evaluate output extraction quality across providers before writing code, run quick tests in the Free Public Chat Playground[cite: 1] or compare models side-by-side in the Model Evaluator[cite: 1].

Conclusion

Building a WhatsApp expense tracker using multimodal vision models eliminates manual data entry and turns receipt photos into structured finance logs in seconds. With Velona's unified REST API gateway, Indian developers can build vision-powered tools funded seamlessly via native UPI top-ups[cite: 1].

Ready to build your multimodal AI apps? Sign up for a free Velona account and start testing today[cite: 1]!