What does robots.txt do?

robots.txt tells web crawlers which pages or sections of your site they are allowed to access. It's a guideline, not enforcement — well-behaved bots respect it, malicious scrapers often ignore it.

Can robots.txt block Google from indexing pages?

Disallowing a URL in robots.txt prevents Googlebot from crawling it, but the URL may still be indexed if other sites link to it. To prevent indexing, use the noindex meta tag or X-Robots-Tag header on the page itself.

robots.txt Generator

Generates a properly formatted robots.txt file for your website. Configure which user-agents (Googlebot, Bingbot, all bots, etc.) can access or are blocked from specific URL paths. Includes the Sitemap URL field and common preset options (allow all, block all, block GPTBot/AI crawlers, protect admin areas). Shows a live preview of the generated file.

Rule 1

User-agent

Crawl-delay (seconds, optional)

Disallow paths (one per line)

Allow paths (one per line)

Sitemap URL (optional)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Configure your crawl rules and copy the generated robots.txt to deploy to your site root.

Frequently Asked Questions

Related Tools

New

Meta Tag Generator

Generate HTML meta tags for SEO, viewport, charset, and more.

New

Open Graph Tag Generator

Generate Open Graph meta tags for social media sharing.

New

Twitter / X Card Generator

Generate Twitter Card meta tags for rich link previews on X (Twitter).

New

UTM Parameter Builder

Build UTM-tagged URLs for campaign tracking in Google Analytics.

New

URL Parser

Parse and break down URLs into their component parts.

New

Keyword Density Analyser

Analyse keyword frequency and density in your content.