What is Robots.txt?

A file on a website that instructs search engine crawlers which pages or folders to index or ignore; misconfiguring it can accidentally block an entire site from Google.

From 9 Roadblocks That Could be Harming Your Google Rankings by Barrie Smith #301

A text file that tells search engines which pages or sections of a site to block from crawling and indexing, used here to prevent development sites from being indexed.

From #SEOpodcast 273 - Common SEO Mistakes You Can Avoid

A file that blocks Googlebot from crawling specific pages but does not prevent pages already indexed from appearing in search results.

From Myths vs Facts - Internet Marketing - #SEOpodcast 232

A file that controls which parts of a site search engine crawlers can access. The episode notes that blocking JavaScript or CSS in this file can harm how Google renders and indexes content.

From Preventing Your Campaign From Failing - #SEOpodcast 256

A text file placed on a server that tells search engines where to go, where not to go, and how often to crawl, with no purpose other than controlling search engine behavior.

From What is a good redirect vs. a bad redirect, change your webpage or website without losing Google juice - Unknown Secrets of SEO E-Webstyle Number 26

A file that signals to the Google bot which pages to index; a misconfigured file can exclude an entire website from Google's index.

From SERP and SEVO - #seopodcast 163