Technical Seo
Crawl Budget
The number of pages a search engine bot will crawl on your site within a given timeframe. It's determined by crawl rate limit (how fast the bot can go without hurting your server) and crawl demand (how much Google actually wants to crawl).
Why Crawl Budget Matters for SEO
If your crawl budget gets wasted on junk pages, your important content takes longer to get discovered and indexed. For large sites with thousands of URLs, poor crawl budget management means new pages can sit undiscovered for weeks.
How Crawl Budget Works
Googlebot allocates a budget based on your server health and the perceived value of your URLs. Pages that return errors, have duplicate content, or sit behind infinite URL parameters eat into that budget. You manage it by blocking low-value URLs and keeping your site architecture clean.
Common Mistakes
- Obsessing over crawl budget on a small site with under 10,000 pages
- Leaving thousands of faceted navigation URLs crawlable
- Not monitoring server log files to see actual crawl behaviour
Want to go deeper?
Read the full guide: Crawl Budget →
Sources & Further Reading: