A user agent is blocked
Review the relevant User-agent, Allow and Disallow rules. A restriction may be intentional; do not remove it without considering how you want that service to use your content.
Crawler access tool · free · no signup
Can AI crawlers access your website?
See which common crawler user agents your robots.txt file allows to reach the homepage. Permission does not mean a crawler will visit or that an AI answer will mention the site.
Individual crawler permissions
Homepage noindex warning
Readable robots.txt rules
How to use the AI crawler checker
- Enter your website domain to inspect its root robots.txt rules against the homepage URL.
- Review each named user agent separately. Different agents may have different permissions.
- If a restriction is unintended, review the matching rule with your developer, publish the change, and check again.
Understand your results and what to fix next
A user agent is allowed
This means the inspected robots rules do not block that agent at the homepage. Firewalls, authentication and other server controls may still prevent access. It does not prove an actual crawler visited.
Homepage has a noindex warning
Review whether the page is meant to appear in search. Robots crawl permissions and indexing directives serve different purposes; changing one does not automatically change the other.
Common questions
Does this check every AI crawler and every URL?
No. The results cover the listed user agents at the homepage URL. Rules can differ on other paths, and the list is not an exhaustive inventory of AI services.
Should I allow every listed bot?
Not necessarily. Decide which services and uses you want to allow. This tool reports permissions; it does not recommend removing deliberate privacy or content-use restrictions.
Technical readiness is only the first step
Sikwati tracks the customer questions you choose through web-search-enabled OpenAI and Gemini API checks, saving dated mentions, links and competitor evidence over time.