All checks › Discoverability
robots.txt
Why it matters
Without robots.txt an agent has nowhere to read the rules of the site or find the sitemap. It is the most basic foundation of discovery: everything else depends on this file existing and being well formed.
Want to see how your site does?
The free scan runs this and every other check in about 30 seconds and shows the evidence for each one.
Scan my site for free