What is a robots.txt file (and how to use it)?

The robots.txt file is a small text file at the root of your site that gives search-engine crawlers instructions on what they may or may not crawl. It's simple but powerful — and easy to get wrong.

What it does

• •Tells crawlers which areas to skip (like admin pages).
• •Points them to your sitemap.
• •Manages crawl activity on large sites.

The common mistake

The biggest robots.txt danger is accidentally blocking pages (or your whole site) from being crawled — which stops them ranking entirely. If traffic mysteriously vanishes, robots.txt is one of the first things to check. On a well-built site it's configured correctly from the start.