SmartKitio

Robots.txt Generator

Create a simple robots.txt that tells crawlers which paths to skip and where your sitemap is.

Your input stays in your browser. Nothing is uploaded.

About this tool

The file starts with User-agent: * so the rules apply to all crawlers, then lists one Allow or Disallow line for each path you enter, and a Sitemap line if you provide one. robots.txt controls crawling, not indexing: a disallowed URL can still appear in search results if other pages link to it. To keep a page out of results, allow it to be crawled and use a noindex robots meta tag instead. The file must be served at the root, for example https://example.com/robots.txt.

How to use it

  1. List paths to block, one per line (for example /admin/).
  2. Optionally list paths to allow inside a blocked folder.
  3. Add your sitemap's full URL.
  4. Save the output as robots.txt at the root of your domain.

Examples

  • Block internal search

    Disallow: /search stops crawlers from requesting internal search results pages.

  • Allow everything

    Leave Disallow empty. The file then only points crawlers to your sitemap.

FAQ

Does Disallow remove a page from Google?
No. It stops crawling. Use a noindex meta tag, which requires the page to be crawlable, to remove it from results.
Can I target specific bots?
This generator writes rules for all user agents. Add separate User-agent groups by hand for specific crawlers.
Is robots.txt a security measure?
No. The file is public and well-behaved crawlers follow it voluntarily. Protect private pages with authentication.