llms.txt Checker & AI Bot Access Test
Enter a domain to see whether it publishes an llms.txt, and whether its robots.txt lets AI crawlers like GPTBot, ClaudeBot and PerplexityBot read it. You get copy-paste fixes and a generated llms.txt template.
The AI crawlers this tool checks
Each one reads your robots.txt separately.
| User-agent | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls pages to train and ground OpenAI models |
| OAI-SearchBot | OpenAI | Indexes pages so they can appear in ChatGPT Search |
| ChatGPT-User | OpenAI | Fetches pages live when a ChatGPT user asks about them |
| ClaudeBot | Anthropic | Crawls pages for Claude |
| PerplexityBot | Perplexity | Indexes pages so Perplexity can cite them |
| Google-Extended | Controls use of your content in Gemini and AI grounding |
FAQ
llms.txt and AI crawler questions
What is llms.txt?
llms.txt is a proposed convention: a Markdown-formatted text file at the root of your site (yourdomain.com/llms.txt) that gives large language models a concise summary of what your site is about and links to its most useful pages. Think of it as a curated, human-written map for AI agents.
How do I add llms.txt to my website?
Create a plain-text file named llms.txt with an H1 title, a one-paragraph summary in a blockquote, and sections of Markdown links to your key pages. Upload it to the root of your site so it is served at /llms.txt. This checker generates a starting template from your homepage that you can edit and publish.
Does llms.txt improve my visibility in ChatGPT or Gemini?
No AI vendor has committed to using llms.txt as a ranking signal, so treat it as cheap, low-risk hygiene rather than a guaranteed win. What reliably matters is that AI crawlers are allowed to read your site, that your pages answer buyer questions clearly, and that trusted third-party sources mention you.
Which AI bots should I allow in robots.txt?
If you want to appear in AI answers, allow GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and Google-Extended (Gemini). OAI-SearchBot and PerplexityBot matter most for being cited in live AI search results.
What happens if I block GPTBot?
Blocking GPTBot stops OpenAI from crawling your pages for model training. ChatGPT Search uses a separate crawler, OAI-SearchBot, so you can block one and allow the other. This tool shows the status of each bot separately.