Build a valid robots.txt file visually. Control crawler access, block AI bots, set crawl delays, and add sitemaps — with a live syntax-highlighted preview.
A robots.txt file is a plain-text file at the root of your domain that tells search engines and other automated crawlers which parts of your site they may request. This generator lets you build a valid, correctly-formatted robots.txt visually — choosing user-agents, adding Allow and Disallow paths, setting crawl-delay, and listing sitemaps — with a live syntax-highlighted preview so you never have to memorise the syntax.
It's especially handy for the rules people most often get wrong. People use it to:
Everything runs local/in-browser — your rules never leave your device, there's no upload step, and no account is required. Build, preview, validate, then copy or download the finished file.
* for all bots or a specific name like Googlebot./admin/), and set an optional crawl-delay per block.yourdomain.com/robots.txt.It's a plain-text file at the root of your domain that gives instructions to web crawlers about which URLs they may request. It's part of the voluntary Robots Exclusion Protocol — well-behaved bots like Googlebot and Bingbot read it before crawling and obey it, but it is advisory, not a security control. Anyone or any bot that ignores the protocol can still request the listed paths.
User-agent names the crawler a block of rules applies to (* matches all bots). Disallow lists a path prefix the bot should not request — Disallow: /admin/ blocks everything under /admin/, and Disallow: / blocks the whole site. Allow creates an exception, re-permitting a path inside an otherwise disallowed area, e.g. Disallow: /assets/ with Allow: /assets/public/.
It must live at the root of your domain so it loads at https://yourdomain.com/robots.txt — crawlers only check that exact location and ignore copies in subfolders. Upload it to your site's public root (often public/, www/, or public_html/). Each subdomain and protocol needs its own file. Platforms like Shopify and Squarespace manage it through their own settings instead.
The Sitemap directive points crawlers to your XML sitemap so they can discover your pages faster. It is independent of any User-agent block, must use a full absolute URL (e.g. Sitemap: https://yourdomain.com/sitemap.xml), and you can list several lines for multiple sitemaps. This generator places them at the end of the file, which is the conventional location.
Crawl-delay asks a bot to wait the given number of seconds between requests, which can ease load on a small or shared server. Bing and Yandex honour it, but Google ignores it — set Googlebot's crawl rate in Google Search Console instead. Use it sparingly: a high delay slows how quickly your pages are crawled and updated in search results.
Only crawling. Disallow tells a bot not to fetch a page, but Google can still index a blocked URL — showing it with no description — if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex meta tag or X-Robots-Tag HTTP header. Disallowing in robots.txt actually prevents Google from ever seeing that noindex instruction.
Partly. The Block AI Bots preset adds Disallow rules for GPTBot (OpenAI), ChatGPT-User, CCBot (Common Crawl), anthropic-ai, and Claude-Web — the main scrapers used to train large language models. Ethical operators honour these rules, but robots.txt is voluntary, so some scrapers ignore it. For content you must protect, combine it with server-side rate limiting or authentication.