robots-patch.txt

The robots-patch.txt file contains robots.txt directives specifically for AI engine crawlers.

What it is

A set of User-agent and Allow/Disallow rules for AI-specific crawlers. These directives tell AI engines which parts of your site they can access.

Why it matters

Many sites accidentally block AI crawlers through overly broad robots.txt rules. robots-patch.txt provides explicit allow rules for the major AI crawlers, ensuring your content is accessible to AI engines.

AI crawlers covered

Crawler	Engine
`GPTBot`	OpenAI (ChatGPT)
`ClaudeBot`	Anthropic (Claude)
`PerplexityBot`	Perplexity
`Google-Extended`	Google AI (Gemini, AI Overviews)
`anthropic-ai`	Anthropic (alternative)

Example output

# AI Crawler Access Rules
# Generated by AEOrank

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: anthropic-ai
Allow: /

How to deploy

Append the contents to your existing robots.txt file:

cat robots-patch.txt >> public/robots.txt