robots.txt is a plain-text file at your domain’s root that gives crawling instructions to well-behaved bots — it’s a request, not an enforceable restriction.
What it can do
- Tell compliant crawlers which paths not to crawl
- Point to your XML sitemap location
- Set crawl-delay preferences for crawlers that respect it
What it can’t do
- Guarantee a page is never indexed — a disallowed-but-linked-to page can still appear in search results without its content shown, since Google can still learn the URL exists from external links; a
noindexmeta tag (which requires the page to be crawlable) is the correct tool for true exclusion from search - Stop malicious or non-compliant bots, which simply ignore it entirely
- Password-protect or truly hide sensitive content — it’s publicly readable by anyone, including attackers looking for interesting disallowed paths
FAQ
Should I disallow my admin login page in robots.txt?
Be cautious — doing so publicly advertises the exact path in a file anyone can read; proper authentication is the actual protection, not obscurity via robots.txt.