HTTP / Web
Queries a remote service
robots.txt Viewer
View and analyze a site's robots.txt file: User-agent directives, Allow/Disallow, sitemaps and crawl controls.
How it works
The robots.txt file, placed at the root, gives indexing robots their crawling rules: forbidden paths (Disallow), allowed paths (Allow) and the location of the sitemap.
It is a convention, not an access control: a malicious robot can ignore it. To truly protect a section, use authentication and server-side controls.
Examples
Frequently asked questions
Where should robots.txt be placed?
At the domain root: https://example.com/robots.txt (one file per domain).
Can a specific page be blocked?
Yes with Disallow, but a blocked page can remain indexed if external links point to it: use noindex to remove a page from the index.