Free · No sign-up required

robots.txt tester & validator

We’ll read your site’s robots.txt and show its rule groups, Sitemap lines and risky rules. Type a URL to instantly see which bots can access it.

We read the robots.txt file in your domain’s root directory; you can then test URLs and edit the file.

How does robots.txt work?

robots.txt is a file in your site’s root directory that tells bots which paths they may crawl. Google applies the longest (most specific) rule that matches a URL; in a tie, Allow wins.

robots.txt doesn’t remove pages from the index; it only blocks crawling. If you don’t want a page to appear on Google, use meta robots noindex.

  • User-agent: which bot the rules apply to
  • Disallow and Allow: blocked and allowed paths (* and $ supported)
  • Sitemap: your XML sitemap URL
  • Rule test: a URL’s status for Googlebot, Bingbot, GPTBot and other bots

Frequently asked questions

What does Disallow: / mean?
Blocks the bot from crawling the entire site. In the User-agent: * group, all bots are blocked and the site may drop out of search results over time.
Should I block CSS and JS files?
No. Google renders pages like a browser; if CSS and JS are blocked, it can’t understand the page correctly and may see mobile usability issues.
When do changes take effect?
Google usually caches robots.txt for up to 24 hours. You can request a recrawl from the robots.txt report in Search Console.

Other free tools

Redirects & HTTP headers301/302 chains, HTTPS and security headers XML sitemap checkerURL count, errors and lastmod Bulk status code checkHTTP status and redirects for up to 20 URLs Hreflang checkerLanguage tags for multilingual sites SEO analysis80+ checks, category scores and prioritized recommendationsPopular SEO site auditTechnical and on-page checks for up to 15 pagesPopular

All free tools

See where your site stands in 20 seconds.

Find your technical issues and AI visibility gaps with a free analysis, then leave the fixing to AutoAiSeo.

  • No card required
  • 5-minute setup
  • Cancel anytime
Analyze for free