This is an intelligence module — it goes deeper than the basic presence checks in the
SEO Core Audit, analysing the actual content of your
robots.txt rules, how many URLs your sitemap declares, and whether a
security.txt file is present for responsible disclosure.
🌿 Beginner — Plain English explanations
Beyond "does it exist" — what it actually says
The SEO Core Audit module confirms your robots.txt and sitemap exist and are reachable. This module goes a level deeper: it reads what your crawl rules actually say, so you can catch mistakes that would otherwise sit unnoticed — like an entire section of your site accidentally blocked, or a sitemap that's technically present but nearly empty.
What's being analysed
- Specific disallow rules — which folders or URL patterns are blocked, and whether any of them are blocking content you actually want indexed (a surprisingly common accident after a site migration or redesign).
- Sitemap URL count — how many pages your sitemap actually lists. A sitemap with far fewer URLs than your site's real page count usually means new content isn't being added to it automatically.
- Sitemap cross-reference — whether your sitemap.xml is actually referenced inside robots.txt, and whether the two agree on the same URL — a mismatch here is easy to introduce and easy to miss.
- security.txt — a small text file telling security researchers how to responsibly report a vulnerability they find on your site.
- security.txt Canonical field — a required field (per RFC 9116) pointing back to the file's own canonical location.
- security.txt Expiry — whether the file's stated expiry date has already passed — an expired security.txt is treated as stale by convention, even if it's technically still there.
- ads.txt — the Authorized Digital Sellers file, relevant for any site running programmatic ads; a missing or malformed one means ad exchanges silently ignore your inventory.
How to fix common issues
If specific sections are blocked unintentionally, remove the offending Disallow line. If your sitemap's URL count looks low, most CMS platforms (WordPress with an SEO plugin, Shopify, custom static sites) can regenerate it automatically on every publish — worth confirming that's actually configured rather than relying on a sitemap generated once and forgotten. A security.txt file takes minutes to add: a plain text file with your security contact, canonical URL, and expiry date, placed at /.well-known/security.txt.