Crawler info

SikwatiBot — our web crawler

You probably found this page because SikwatiBot showed up in your server logs. SikwatiBot is the crawler that powers Sikwati, a Generative Engine Optimization (GEO) tool used by site owners and agencies to track how AI engines cite public websites. Authenticated customers save page URLs in their workspace; DNS ownership verification is not required. We respect robots.txt, do not sign in to target sites, and do not edit their pages.

User agent

All SikwatiBot requests use this exact User-Agent header:

Mozilla/5.0 (compatible; SikwatiBot/1.0; +https://sikwati.app/bot)

The +https://sikwati.app/bot token is the convention for identifying friendly crawlers — this page is where it points.

What it fetches

  • /robots.txt — to honor your crawl preferences.
  • /sitemap.xml and common sitemap paths — to discover pages.
  • Public HTML pages on your saved domain (up to 200 per site in the full product, three per staging audit) — to score them for GEO readiness.

We follow at most 5 redirects per request, time out after 10 seconds, and never download more than 5 MB per response. We do not execute JavaScript, do not log in, and never POST.

Verifying it's actually us

SikwatiBot does not currently publish a fixed egress IP range, so User-Agent matching is the recommended way to identify us. The string contains the literal substring SikwatiBot; matching on that is stable across releases. We will publish reverse-DNS validation and a published IP range as we grow.

Allowlist us on Cloudflare

If your site is behind Cloudflare and page discovery fails with a network-layer error, the cause is almost always Bot Fight Mode rejecting our requests. Note that Cloudflare’s legacy User Agent Blocking tool (under WAF → Tools) only blocks — it has no Skip / Allow action. To allow SikwatiBot, use one of the two paths below.

Pro plan or above — Custom rule

  1. Cloudflare dashboard → your domain → Security → WAF → Custom rules
  2. Create rule
  3. Field: User Agent · Operator: contains · Value: SikwatiBot
  4. Then take action: Skip
  5. Skip the following: tick Bot Fight Mode and All managed rules
  6. Deploy. Effective immediately at the edge.

The equivalent expression for the Expression Editor: (http.user_agent contains "SikwatiBot")

Free plan — disable Bot Fight Mode

Free-tier Custom rules don’t support the Skip action against Bot Fight Mode, so the practical option is to disable BFM for the site:

  1. Cloudflare → your domain → Security → Bots (or Security → Settings on older dashboards)
  2. Turn off Bot Fight Mode (and Super Bot Fight Mode if shown)
  3. Save

Or upgrade to Pro and use the Custom rule above — that scopes the allowlist to SikwatiBot only and leaves BFM protecting the rest.

Allowlist on other WAFs

For AWS WAF, Akamai, Fastly, and similar, the rule is identical in spirit: match the substring SikwatiBot in the User-Agent header and bypass bot-management rules. Consult your provider’s docs for the dashboard click-path; the underlying match is the same.

If you do not want SikwatiBot to crawl your site

We honor robots.txt. To block us entirely, add this to https://yoursite.com/robots.txt:

User-agent: SikwatiBot
Disallow: /

This is the standard mechanism — same as for Googlebot or any other well-behaved crawler. SikwatiBot will stop hitting your site within one crawl cycle.

Reporting issues or abuse

If SikwatiBot is misbehaving — not honoring robots.txt, hitting your site too aggressively, or you suspect a User-Agent impersonator — email abuse@sikwati.com with the date, IP, and a sample log line. We treat these reports as P1.