Robots.txt Generator

Rules for all crawlers

Google ignores this setting, but Bing and some others follow it.
Use this to allow a specific file inside a blocked folder.

Rules for specific crawlers (optional)

Choose “Block” to keep a crawler out completely, for example AI training bots. A crawler with its own rule follows only that rule and ignores the rules above.

Everything is generated in your browser and nothing is uploaded or stored.

How to use the robots.txt generator

Choose whether crawlers are allowed in by default, add any folders or files you want to keep them out of, and add your sitemap address. Your robots.txt file updates as you type. Click Copy Code or Download robots.txt, then upload the file to the root of your website so it can be found at yourdomain.com/robots.txt. If you use WordPress, the Load WordPress Preset button fills in the usual settings.

What is a robots.txt file?

A robots.txt file is a small text file that tells search engine crawlers and other bots which parts of your website they may visit. It is one of the first files crawlers look for.

  • User-agent: names the crawler a rule applies to. An asterisk means all crawlers.
  • Disallow: a folder or page the crawler should not visit. A single slash blocks the whole site.
  • Allow: a specific path that is allowed, usually inside a blocked folder.
  • Sitemap: the full address of your XML sitemap, which helps crawlers find your pages.
  • Crawl-delay: asks a crawler to wait between requests. Google ignores it.

Blocking AI and other specific crawlers

You can give individual crawlers their own rules, such as blocking AI training bots while keeping Google and Bing allowed. A crawler that has its own group of rules follows only that group and ignores the general rules. Well-behaved crawlers respect robots.txt, but following it is voluntary.

Important things to know

  • Robots.txt is not a security tool. Anyone can read the file, and blocked pages can still be visited directly, so never use it to hide private information.
  • Blocking a page does not remove it from search results if it is already indexed or linked from elsewhere. To keep a page out of search, use a noindex tag instead, which you can create with our Meta Tag Generator.
  • Avoid blocking CSS and JavaScript files, because search engines need them to display your pages properly.
  • After uploading, test your file with the robots.txt report in Google Search Console.
Scroll to Top