robots.txt Generator

Build a robots.txt that tells search engines what to crawl. Add per-bot rules, crawl delays and sitemaps.

Crawl rules

Allow (one per line)Disallow (one per line)

Crawl delay (sec, optional)

Sitemap URLs (one per line, optional)

3 lines

User-agent: *
Allow: /

Allow all, disallow all, block admin pages or block AI scrapers.

Add rules for specific crawlers with allow/disallow paths, one per line.

Optionally slow down aggressive bots with a crawl-delay in seconds.

List your sitemap.xml URLs so crawlers discover every page.

The robots.txt preview updates instantly as you edit the rules.

Download robots.txt and upload it to your domain root.

Allow all, disallow all, block admin pages or block AI scrapers.

Add rules for specific crawlers with allow/disallow paths, one per line.

Optionally slow down aggressive bots with a crawl-delay in seconds.

List your sitemap.xml URLs so crawlers discover every page.

The robots.txt preview updates instantly as you edit the rules.

Download robots.txt and upload it to your domain root.

FAQ

What is a robots.txt file?

A text file at your domain root that tells search engine crawlers which pages they may or may not visit.

Where do I put the generated robots.txt?

Upload it to your domain root so it is served at https://yourdomain.com/robots.txt.

Do all bots obey robots.txt?

Well-behaved crawlers like Googlebot obey it. It is advisory — malicious scrapers may ignore it.

Can I block specific pages from crawlers?

Yes, add disallow rules for any path, from single pages to whole directories.

Is the robots.txt generator free?

Yes, building and downloading robots.txt files is completely free.