TheSeoSoul
Free tools/Technical

Free Robots.txt Checker

Instant online robots txt checker for any public domain. See whether the file exists, if crawlers are blocked site-wide, which Sitemap URLs are declared, and how common AI bots are treated — free, no signup.

How it works

  1. 01

    Enter a domain or full URL

    We resolve the host and request /robots.txt over HTTPS when available (with a sensible fallback).

  2. 02

    Parse crawl rules and Sitemap lines

    The response body is scanned for User-agent groups, Disallow / Allow directives, and Sitemap: URLs.

  3. 03

    Flag high-impact and AI-bot policy

    We highlight crawl-all blocks and known generative crawler user-agents so you can verify intent at a glance.

What this robots.txt checker reports

Built for technical SEO reviews and quick pre-launch checks — not a full Search Console crawl report.

  • HTTP status and whether robots.txt was found
  • Site-wide Disallow patterns that block all crawlers
  • Sitemap directives listed in the file
  • Presence of rules for popular AI crawlers (e.g. GPTBot, ClaudeBot)
  • Raw file preview so you can audit the exact text crawlers see

Scenarios

Critical Robots.txt Scenarios & Mistakes

Robots.txt syntax errors can invisibly dismantle organic rankings overnight. Common scenarios include:

Crawl Disasters

Accidental Disallow: / in Production

The Problem: Deploying a staging robots.txt with Disallow: / immediately shuts out Googlebot, wiping the domain from SERPs within days.

The Fix: Always verify production robots.txt explicitly specifies Allow: / for User-agent: * and lists the canonical XML sitemap.

Rendering Issues

Blocking CSS & JavaScript Assets

The Problem: Disallowing /_next/static/ or /wp-content/ prevents Googlebot from rendering the DOM, failing Mobile-Friendly and Core Web Vitals audits.

The Fix: Never block access to CSS, JS, fonts, or image assets required to render modern client/server applications.

AI Crawler Policy

Targeted AI Scraper Blocking

The Problem: You want to block automated LLM training scrapers (GPTBot, CCBot) without blocking Google Search or Bing indexation.

The Fix: Target individual bot User-agents (e.g., User-agent: GPTBot Disallow: /) while keeping User-agent: * open for search engines.

Standard Production robots.txt Templates

A clean robots.txt allows search crawlers full access while keeping internal endpoints and unwanted scrapers out:

Production robots.txt Templatetext
# Allow all legitimate search engine crawlers
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

# Optional: Block LLM training crawlers
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

# Canonical XML Sitemap
Sitemap: https://example.com/sitemap.xml
Next.js App Router (app/robots.ts)typescript

Generate your robots.txt dynamically with type safety in Next.js:

import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/api/', '/admin/'],
    },
    sitemap: 'https://example.com/sitemap.xml',
  };
}

Frequently asked questions

What does a free robots.txt checker do?
It fetches the public robots.txt for a domain (usually https://example.com/robots.txt), shows whether the file exists, and highlights high-impact rules such as Disallow: / for all user-agents, Sitemap lines, and blocks aimed at AI crawlers like GPTBot or ClaudeBot.
Is robots.txt the same as noindex?
No. robots.txt tells crawlers which paths they may fetch. Indexing of a URL is also controlled by meta robots and the X-Robots-Tag HTTP header. A page can be crawlable but still noindex — check both.
Why check AI crawler rules in robots.txt?
Many generative engines honor robots.txt for training and browsing bots. Blocking GPTBot, ClaudeBot, or similar agents is a deliberate policy choice. This checker surfaces those user-agent blocks so you can confirm what you intended to publish.
Does this tool submit my site to Google?
No. It only performs a live HTTP fetch of robots.txt from our servers and displays the result. Nothing is submitted to Search Console or any search engine.
Do I need an account?
No. TheSeoSoul robots txt checker is free with no registration. For a full Meta / Structure / Technical / GEO audit of the same domain, use the website SEO checker on the homepage.

Keep going

What to check next

robots.txt only controls crawl access — check indexing and preferred URLs next.

Need the whole picture? Run a full audit · browse every tool.