Robots.txt
A file that instructs search engine crawlers which pages to crawl or avoid.
Definition
Robots.txt is a plain text file placed at the root of a website (domain.com/robots.txt) that provides instructions to web crawlers about which pages or sections they are allowed or disallowed from crawling. It uses Disallow and Allow directives. While robots.txt can prevent pages from being crawled, it does not prevent them from being indexed if those pages are linked from other sites.
Why It Matters
Robots.txt is a fundamental technical SEO tool for managing crawler access and optimizing crawl budget. Blocking low-value URL spaces from crawlers preserves crawl budget for important pages. However, robots.txt errors — accidentally blocking critical JavaScript, CSS, or content pages — are among the most serious and common technical SEO mistakes.