Build an Autonomous AI News Scraper & Digest Bot

Staying updated with industry developments, market trends, or technical releases requires monitoring dozens of blogs, news outlets, and RSS feeds daily. Manual browsing consumes hours and often results in information overload.

In this tutorial, we will build an Autonomous AI News Scraper & Executive Digest Bot using Python, BeautifulSoup, httpx, and Velona's Unified AI Gateway. The bot automatically scrapes headlines from specified news sources, filters duplicate stories, synthesizes key points into a clean Markdown executive summary, and delivers the report to your team.

Architecture of an Autonomous Summarization Pipeline

The autonomous pipeline runs on a scheduled background cadence (e.g., daily at 8:00 AM IST):

Scraping & Extraction: Python fetches HTML content from target industry news outlets or RSS feeds using httpx and parses headlines and article bodies with BeautifulSoup.
Filtering & Deduplication: Standardize text content and drop duplicate story topics before processing.
AI Synthesis: Send extracted text batches to Velona's unified REST API endpoint (/gateway/v1/inference/run)[cite: 1] using ultra-fast, low-cost models.
Digest Formatting: Compile structured Markdown outputs containing key executive takeaways, impact ratings, and direct source links.

Step 1: Set Up Your Velona Account & API Key

To run scheduled background summarization jobs across 300+ LLMs[cite: 1]:

Register on the Velona Sign-Up Page[cite: 1].
Go to your Prepaid Wallet Dashboard and top up your account balance starting at ₹10 using UPI (GPay, PhonePe) or RuPay[cite: 1]. Velona processes transactions natively in Indian Rupees without foreign currency conversion markups[cite: 1].
Navigate to API Key Management and create an API key tagged for news-scraper-bot[cite: 1].
Set an optional monthly spending limit in paise on the key to protect your wallet balance during high-volume scraping jobs[cite: 1].

Step 2: Install Python Scraping Dependencies

Open your terminal and install the required scraping and HTTP libraries:

pip install httpx beautifulsoup4 requests

Step 3: Build the Autonomous Scraper & Summarizer

Create a Python script named news_digest_bot.py:

import httpx
import requests
from bs4 import BeautifulSoup

VELONA_API_KEY = "YOUR_VELONA_API_KEY"
VELONA_URL = "https://velona.in/gateway/v1/inference/run"

headers = {
    "Authorization": f"Bearer {VELONA_API_KEY}",
    "Content-Type": "application/json"
}

# 1. Scrape Articles from News Sources
def fetch_latest_tech_headlines():
    url = "https://news.ycombinator.com"
    response = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"})
    soup = BeautifulSoup(response.text, "html.parser")
    
    articles = []
    titles = soup.find_all("span", class_="titleline")[:10]
    
    for idx, title_tag in enumerate(titles, 1):
        link_tag = title_tag.find("a")
        if link_tag:
            articles.append(f"{idx}. {link_tag.text} ({link_tag['href']})")
            
    return "\n".join(articles)

# 2. Synthesize Executive Digest via Velona Gateway
def generate_ai_digest(raw_headlines: str):
    system_prompt = """You are an executive intelligence analyst for a tech enterprise.
Review the provided raw news headlines and generate a structured Daily Executive Digest in Markdown.

Include:
1. **Top 3 Industry Trends**: High-level synthesis of major movements.
2. **Key Actionable Takeaways**: 1-sentence bullet points summarizing crucial technical or business news.
3. **Sentiment & Impact**: Brief assessment of overall market impact.

Keep the output concise, executive-ready, and formatted cleanly."""

    payload = {
        "model": "google/gemini-flash-1.5",
        "turns": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": f"Raw Headlines:\n{raw_headlines}"}
        ]
    }

    res = requests.post(VELONA_URL, json=payload, headers=headers)
    result = res.json()
    return result["choices"][0]["message"]["content"]

# Execution Loop
if __name__ == "__main__":
    print("--- Scraping Latest Industry Headlines ---")
    headlines = fetch_latest_tech_headlines()
    print(f"Extracted {len(headlines.splitlines())} stories.\n")
    
    print("--- Synthesizing AI Executive Digest ---")
    digest = generate_ai_digest(headlines)
    
    print("\n================ DAILY EXECUTIVE DIGEST ================\n")
    print(digest)

Cost-Effective Model Selection for High-Volume Summaries

Scraping and summarizing long articles daily requires balancing processing speed with low per-token expenses[cite: 1]:

Model	Ideal Use Case	Cost / Speed Benefit
`google/gemini-flash-1.5`	Daily high-volume news parsing and batch summarization	Ultra-low cost per token with massive context processing capabilities[cite: 1].
`meta/llama-3.1-70b-instruct`	Filtering noise, classifying topics, and deduplication	High throughput (tokens/sec) for rapid text extraction[cite: 1].
`anthropic/claude-3.5-sonnet`	Weekly deep-dive strategic analysis and market reports	Superior language nuance and high-level strategic reasoning[cite: 1].

You can check real-time per-token prices in Indian Rupees on the Velona Model Pricing Index[cite: 1] and track token consumption for your background workers on the Usage Analytics Dashboard[cite: 1].

Developer Utilities & Workbench Tools

When parsing scraped HTML strings, formatting Markdown newsletters, or validating JSON payloads, make use of Velona's 49 Life Time Free Developer Tools[cite: 1, 1]. Utilities like the Markdown Editor, HTML Entities Converter, and Regex Evaluator run 100% free without deducting funds from your wallet[cite: 1].

To test system prompt structures before deploying scheduled background scripts, experiment in the Free Public Chat Playground[cite: 1] or evaluate summary outputs side-by-side using the Model Evaluator[cite: 1]. You can also benchmark reasoning capabilities on the AI Benchmark Leaderboard[cite: 1].

Conclusion

Building an autonomous news scraper and AI digest bot cuts through online noise, delivering high-signal market intelligence straight to your team. With Velona's unified REST API gateway, Indian developers and startups can automate daily AI data workflows funded easily via local UPI top-ups[cite: 1].

Ready to automate your intelligence pipelines? Sign up for a free Velona account and start building today[cite: 1]!

Build an Autonomous News Digest & Web Scraper Bot with AI