Free tool
Robots.txt Generator
Check your existing robots.txt file and create a clean, Google-friendly version with the right crawl and sitemap directives.
- Instant results
- No signup
- Public URLs only
How it works
Step 1
Paste your domain
We fetch /robots.txt and parse it with longest-match rules, the way Google does.
Step 2
See what it really blocks
Not what it looks like it blocks. The winning rule is quoted for every verdict.
Step 3
Take the clean version
With your sitemap, and an explicit line on AI crawlers instead of an accidental one.
Questions
- I do not have a robots.txt. Is that bad?
- No. Absence is permission — a site with no robots.txt allows everything. Publishing one is how you say something other than 'everything'.
- Should I block AI crawlers?
- It is a decision, not a default, and the point of this tool is to make it one you took. Marketing copy is written to be repeated; a pricing page quoted in an assistant's answer is a visit. Private surfaces are a different question.
- Does robots.txt keep a page out of Google?
- No — it stops the fetch, not the indexing. A disallowed URL with links pointing at it can still appear, URL-only. Use noindex for that, and do not disallow the page, or the crawler never reads the tag keeping it out.
Other free tools
llms.txt Generator
Generate a spec-compliant llms.txt for your site in seconds. Free, no signup, ready to publish.
Meta Tag Checker
Check any page's title, description, canonical, robots and social tags. Free, instant.
Schema Markup Generator
Build valid JSON-LD schema for 12 page types, with live validation. Free, in your browser.
Can assistants actually read your site?
The AI visibility audit knocks as ChatGPT, Claude, Gemini and Perplexity, then asks them what they say about you. Free, and it runs in about two minutes.
No card. One per address.