Technical Seo
Robots.txt
A plain text file placed at the root of your domain that tells search engine crawlers which pages or sections of your site they're allowed or disallowed from accessing.
Why Robots.txt Matters for SEO
It's your first line of defence for controlling what gets crawled. A misconfigured robots.txt can block your entire site from indexing or waste crawl budget by leaving low-value areas wide open.
How Robots.txt Works
Crawlers check yoursite.com/robots.txt before crawling. You specify User-agent directives for different bots and Disallow rules for paths you want blocked. It's advisory, not enforcement. Well-behaved bots respect it, but it doesn't prevent indexing if pages get links from elsewhere.
Common Mistakes
- Using Disallow to try to deindex pages instead of noindex
- Blocking CSS and JS files that Googlebot needs to render pages
- Leaving a staging site's Disallow: / in place after going live
Want to go deeper?
Read the full guide: Robots.txt →
Sources & Further Reading: