Skip to content
Kordu Tools
SEO & Web Runs in browser Updated 02 Apr 2026

Robots.txt Generator

Generate and validate your robots.txt file with a form builder, CMS presets, and instant error checking.

CMS Presets

Rules

Note: Googlebot ignores Crawl-delay. Use Google Search Console instead.

Sitemap URLs

Preview
Loading rating…

How to use Robots.txt Generator

  1. Add crawl rules

    Choose a User-agent (or use * for all bots) and add Allow or Disallow directives for specific paths like /admin/ or /checkout/.

  2. Apply a CMS preset

    Click a preset button for WordPress, Shopify, or Next.js to auto-fill common crawl rules for that platform, including admin paths and duplicate content patterns.

  3. Add your sitemap URL

    Enter your sitemap URL (e.g. https://example.com/sitemap.xml) so search engines can find and index your content efficiently.

  4. Review the live preview

    The generated robots.txt file updates in real time as you make changes. Check it carefully before copying.

  5. Copy or download

    Click Copy to clipboard or Download as robots.txt and upload the file to the root of your domain.

Robots.txt Generator FAQ

What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages of your site they can and cannot access. It lives at the root of your domain (e.g. https://yourdomain.com/robots.txt) and is checked by crawlers before they begin indexing.

Which User-agent should I use?
Use * (asterisk) to apply rules to all crawlers. Add specific rules for individual bots — Googlebot, Bingbot, GPTBot (OpenAI), CCBot (Common Crawl), and others — if you need different behaviour per crawler.

Does Disallow remove pages from Google's index?
No. Disallow prevents crawling but does not remove already-indexed pages. To deindex a page, use a noindex meta tag or the URL Removal Tool in Google Search Console.

What is Crawl-delay?
Crawl-delay tells crawlers how many seconds to wait between requests to reduce server load. Google ignores this directive — use Google Search Console to control Googlebot's crawl rate instead. Other crawlers like Bingbot do respect it.

Should I block my admin and login pages?
Yes. Paths like /admin/, /wp-admin/, /login/, /dashboard/, and /checkout/ should be blocked to prevent crawlers from wasting crawl budget on pages that should never be indexed.

What does the validator check for?
The validator checks for syntax errors, conflicting Allow/Disallow rules for the same path, missing Sitemap directives, unusually high Crawl-delay values, and unknown or misspelled directives.

What is the difference between Allow and Disallow?
Disallow blocks a crawler from accessing a path. Allow explicitly permits a path even if a broader Disallow rule covers it. Allow takes precedence over Disallow for the same path.

Is my robots.txt file sent to a server?
No. All generation and validation happens in your browser. Your file content is never transmitted to any server.

How do I block AI training bots?
Add rules for specific AI crawlers: Disallow: / for GPTBot (OpenAI), CCBot (Common Crawl), anthropic-ai, and Google-Extended. The WordPress and Next.js presets include these by default.

Background

Your robots.txt file is the first thing search engine crawlers read when they visit your site. Getting it wrong can block your entire site from Google or waste crawl budget on pages that should never be indexed.

This tool gives you a visual form builder for robots.txt — no need to memorise directives or worry about syntax. Add rules for any search engine crawler (or use * for all bots), set Allow and Disallow path patterns, declare your Sitemap URL, and apply Crawl-delay settings. One-click CMS presets auto-fill sensible defaults for WordPress, Shopify, and Next.js, covering admin paths, login pages, duplicate content patterns, and bot-specific rules for GPTBot and other AI crawlers.

The Validate mode lets you paste an existing robots.txt and check it for syntax errors, conflicting Allow/Disallow rules, missing Sitemap directives, overly aggressive Crawl-delay values, and unknown directives. All checking runs in your browser with no file upload required.

Who is this for? SEO specialists auditing crawl settings, developers deploying new sites, e-commerce managers protecting admin and checkout paths, and content teams managing crawl budget on large sites.

Related tools

Meta Tag Previewer

Preview how your page looks in Google, Facebook, Twitter, and LinkedIn before you publish. Generate ready-to-paste meta tags.

Schema Markup Generator

Generate and validate JSON-LD structured data for 10 schema.org types. Unlock rich results in Google search — free, browser-based.

XML Sitemap Generator

Generate sitemap.xml from a URL list or validate your existing sitemap — checks priority, changefreq, lastmod, and duplicates.

Regex Tester

Test regular expressions live with color-coded match highlighting, capture groups, replace mode, and common presets.