For site owners
About the AEOReadinessAudit crawler
Our crawler fetches a single product page when someone asks us to audit it, so we can tell them how ready that page is for AI shopping engines. It doesn't crawl your site, follow links or collect data for AI training.
How to recognize it
Every request sends this user agent:
Mozilla/5.0 (compatible; AEOReadinessAudit/0.1; +https://aeo.damonpadron.com/bot)
What it requests
- The product page URL that a user submitted, once per audit.
/robots.txtand/llms.txt, to report which crawlers your rules allow.- Sometimes a headless-browser render of the same page, to check which content depends on JavaScript. Images, fonts and media are not downloaded.
If you return a 429 we wait and retry at most twice, respecting Retry-After up to 10 seconds. Audits are started by people, one page at a time; there is no background crawling.
robots.txt
Each fetch is a one-off request made on behalf of a person, so generic rules (User-agent: *) don't apply to it, in the same way they don't apply to a browser. Rules that name our crawler do: add User-agent: AEOReadinessAudit with Disallow: / and we won't fetch your pages. We also read your robots.txt to report on other crawlers (OAI-SearchBot, PerplexityBot, Googlebot and others) and never impersonate them.
Allowing it
If your bot protection blocks audits of your own pages, add a rule that skips bot challenges for requests whose user agent contains AEOReadinessAudit. In Cloudflare, that's a WAF custom rule with the action Skip. User agents can be spoofed, so scope the rule to your product paths if you prefer.
You don't have to allow it: the audit form also accepts pasted page source.
Blocking it or reporting a problem
Blocking the user agent at your firewall stops all audits of your site. To ask us to exclude your domain or report a problem, email padrondamon@gmail.com.