ShopiensBot

Every request Shopiens sends to a customer's site carries this exact User-Agent, verbatim:

Mozilla/5.0 (compatible; ShopiensBot/1.0; +https://shopiens.ai/bot; formerly EvaristBot)

The product token is ShopiensBot. Match it in a robots.txt group, a WAF or CDN bot rule, or your access logs. The version number identifies crawler behaviour, not a release; it changes only when what we do changes.

The string still names EvaristBot, our crawler's former name, so a WAF or CDN rule you wrote for it keeps matching for now. Move that rule to ShopiensBot: EvaristBot will leave the string.

We read robots.txt first and obey it for every page we render; if a registered domain's robots.txt can't be read, we treat it as allowing everything; a subdomain we only found through your links is skipped until its own robots.txt has been read. Your homepage renders next, in a real browser, so client-rendered sites read correctly and not just their initial HTML. Then we read sitemap.xml to plan the rest of the crawl. Pages are rendered through Cloudflare's browser rendering service, so page requests come from Cloudflare's network; robots.txt and sitemap requests come from our own servers.

Two other kinds of request carry the same User-Agent but aren't the crawl: checking a no-code tag while an account holder authors it, and a daily tag-health check re-verifying a tag. Each fetches single pages, and robots.txt, covered next, doesn't gate them. The install check makes no request to your site.

• Before crawling, we read robots.txt and obey it, including a group addressed specifically to ShopiensBot, or to EvaristBot, its former name. This governs the crawl. It doesn't gate the tag checks described above, which fetch single pages.
• Normally only one crawl runs against a site at a time.
• Concurrent requests stay low: typically single digits, low double digits at most during a brief overlap between a scheduled crawl and a refresh.
• A crawl covers up to about 150 pages, and scheduled refreshes are full crawls. Adding a domain triggers a smaller crawl of that domain.
• We refresh roughly every couple of weeks, and stop entirely for sites that go quiet.

Match the ShopiensBot token in your WAF's or CDN's bot rules. Cloudflare, Akamai and similar platforms let you allow (or block) a request by its user-agent string. Some platforms, Akamai's custom bot categories for instance, also let you define your own named bot category this way. A pre-built "verified bot" directory, where a platform has one, is typically populated by the bot operator applying to be listed, not something a site owner adds a bot to from their own dashboard, so it isn't where you'll find ShopiensBot; the custom rule above is the one to use.

A user-agent identifies, it doesn't authenticate. Anyone can send this header, so treat a UA rule as "let this crawler through", never as proof of who is calling. Scope it to what you'd let any well-behaved crawler do, which is reading pages you already publish, and don't let it bypass authentication or reach anything you wouldn't serve to the open web. Pages are rendered by Cloudflare's browser service, so page requests come from Cloudflare's network. For a rule that can't be spoofed, email support@shopiens.ai.

We honour a robots.txt rule that disallows us:

User-agent: ShopiensBot
Disallow: /

A group naming ShopiensBot replaces the * group for us entirely; it doesn't add to it. Scope the rules under your ShopiensBot group completely; we won't fall back to whatever the * group also allows or disallows.

A group you wrote for EvaristBot, our former name, still binds us. When groups name both, we follow the rules of all of them together.

This stops the crawl. The tag checks described above aren't gated by robots.txt; to stop those too, block the ShopiensBot token in your WAF or CDN.