HTTP / Web Queries a remote service

robots.txt Viewer

View and analyze a site's robots.txt file: User-agent directives, Allow/Disallow, sitemaps and crawl controls.

How it works

The robots.txt file, placed at the root, gives indexing robots their crawling rules: forbidden paths (Disallow), allowed paths (Allow) and the location of the sitemap.

It is a convention, not an access control: a malicious robot can ignore it. To truly protect a section, use authentication and server-side controls.

Examples

Standard site

Google directives and sitemap.

urlhttps://www.google.com
Load this example

Frequently asked questions

Where should robots.txt be placed?

At the domain root: https://example.com/robots.txt (one file per domain).

Can a specific page be blocked?

Yes with Disallow, but a blocked page can remain indexed if external links point to it: use noindex to remove a page from the index.