Generates a properly formatted robots.txt file for your website. Configure which user-agents (Googlebot, Bingbot, all bots, etc.) can access or are blocked from specific URL paths. Includes the Sitemap URL field and common preset options (allow all, block all, block GPTBot/AI crawlers, protect admin areas). Shows a live preview of the generated file.
Configure your crawl rules and copy the generated robots.txt to deploy to your site root. Complete your SEO setup with the Meta Tag Generator and Open Graph Generator.
Block GPT crawler
Input
Block GPTBot, allow all others
Output
User-agent: GPTBot Disallow: / User-agent: * Allow: /
Generate HTML meta tags for SEO, viewport, charset, and more.
Generate Open Graph meta tags for social media sharing.
Parse and break down URLs into their component parts.
Count characters for page titles, meta descriptions, and social posts with visual limits.
Generate Twitter Card meta tags for rich link previews on X (Twitter).
Build UTM-tagged URLs for campaign tracking in Google Analytics.