# Robots.txt | SEO Glossary

> Robots.txt explained in plain English. Understand what robots.txt means, why it matters for SEO, and how it works.

**Source:** https://www.lawrencehitches.com/glossary/robots-txt/

---

Breadcrumb Category tag Technical Seo H1

# Robots.txt

Definition block A plain text file placed at the root of your domain that tells search engine crawlers which pages or sections of your site they're allowed or disallowed from accessing.

---

Expanded explanation

## Why Robots.txt Matters for SEO

It's your first line of defence for controlling what gets crawled. A misconfigured robots.txt can block your entire site from indexing or waste crawl budget by leaving low-value areas wide open.

## How Robots.txt Works

Crawlers check yoursite.com/robots.txt before crawling. You specify User-agent directives for different bots and Disallow rules for paths you want blocked. It's advisory, not enforcement. Well-behaved bots respect it, but it doesn't prevent indexing if pages get links from elsewhere.

## Common Mistakes

- Using Disallow to try to deindex pages instead of noindex
- Blocking CSS and JS files that Googlebot needs to render pages
- Leaving a staging site's Disallow: / in place after going live

Video (if available) Article link (if exists) Want to go deeper? Read the full guide: Robots.txt → External sources Sources & Further Reading:

- [Google: Robots.txt Introduction and Guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [Google: Create and Submit a robots.txt File](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt)

Related articles Related articles:

- [LLM SEO: Rank in AI Search Engines](https://www.lawrencehitches.com/llm-seo/)
- [Where Does ChatGPT Get Its Data From?](https://www.lawrencehitches.com/where-does-chatgpt-get-its-data-from/)

Related terms Author bio About the Author

Lawrence Hitches is an [AI SEO consultant based in Melbourne](https://www.lawrencehitches.com/) and Chief of Staff at StudioHawk. He specialises in AI search visibility, technical SEO, and organic growth strategy. [Book a free consultation →](https://www.lawrencehitches.com/contact/)

---

*Lawrence Hitches is an AI SEO consultant in Melbourne and Chief of Staff at StudioHawk, Australia's most awarded SEO agency. Agent edition of this site: https://www.lawrencehitches.com/md/index.md*