Robots.txt Tester & Analyzer
Fetch any domain's robots.txt and see it parsed: which user-agents are addressed, what each may and may not crawl, and whether sitemaps are declared. One wrong line here can deindex a site, worth checking twice.
How to use
- Enter the domain (the tool fetches https://domain/robots.txt).
- Click Run to fetch and parse the rules.
- Review each user-agent group's Allow/Disallow rules and the Sitemap lines.
robots.txt is a plain-text file at a site's root that tells crawlers which paths they may fetch. It is the first thing a well-behaved crawler requests, and it is governed by a handful of rules that are widely misunderstood, misunderstandings that have deindexed entire businesses.
The essentials: rules group under User-agent lines (* for everyone, or a specific bot like Googlebot). Disallow blocks path prefixes and Allow carves exceptions. The most specific matching rule wins. Sitemap: lines advertise your XML sitemap to every crawler without waiting for manual submission. An empty Disallow: means 'allow everything', while Disallow: / blocks the entire site, a single character of difference that has shipped to production more than once.
The critical misconception: robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in Google results (as a bare URL with no description) if other sites link to it, and because the crawler cannot fetch the page, it can never see a noindex tag on it. To remove a page from search, you must allow crawling and serve noindex. Blocking it in robots.txt actually prevents removal. Also remember robots.txt is public and advisory, never use it to 'hide' admin paths (it is a signpost to them) and never expect malicious bots to obey it.
This analyzer flags the classic mistakes: blocking CSS/JS (which breaks Google's rendering), missing sitemap lines, and rules that contradict each other.
Examples
Input: velona.in
Output: User-agent: * · Disallow: /admin, /dashboard, /api/ · Sitemap: https://velona.in/sitemap.xmlInput: staging.example.com
Output: User-agent: * · Disallow: /, WARNING: entire site blocked from all crawlersFrequently asked questions
Does blocking a page in robots.txt remove it from Google?
No, often the opposite. Blocked pages can still be indexed from external links, shown as bare URLs, and because Google cannot crawl them it never sees a noindex tag. To deindex a page: allow crawling, add a noindex meta tag or header, and wait for a recrawl.
What is the difference between Disallow: and Disallow: /?
Everything. Disallow: with nothing after it allows all crawling for that user-agent. Disallow: / blocks the entire site. The single slash is one of the most expensive characters in SEO.
Should I put my sitemap in robots.txt?
Yes, a Sitemap: line with the absolute URL lets every crawler discover it automatically, including the ones you never submit to via webmaster consoles. It coexists fine with Search Console submission.
Can I rely on robots.txt to protect private pages?
Absolutely not. It is a public file that merely requests good behavior, and it handily lists your sensitive paths for anyone who looks. Protect private areas with authentication. Use robots.txt only to manage legitimate crawler traffic.
Related tools
Velona is India's INR-native AI API gateway with 300+ models, UPI top-up from ₹10, no foreign card needed. These tools are free forever.