# The Stages of Search Engine Crawling **Last updated:** 2024-07-17 **Source:** https://www.lawrencehitches.com/search-engine-crawling/ --- wp:post-featured-image / wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Crawling is the initial step in a search engine finding your website's content. Without crawling, your site won't be indexed or ranked, resulting in no traffic. /wp:paragraph wp:paragraph Google announced in 2016 that they have knowledge of [130 trillion pages](https://searchengineland.com/googles-search-indexes-hits-130-trillion-pages-documents-263378), a process that is remarkably complicated and completely automatic. /wp:paragraph wp:paragraph However, research from Ahrefs shows that [96.55% of content](https://ahrefs.com/blog/search-traffic-study/) receives zero traffic from Google. This indicates that search engines like Google only crawl and index select content. /wp:paragraph wp:paragraph Search engine crawling is an essential process that helps search engines explore and index websites. /wp:paragraph wp:paragraph To grasp how this works, it's important to understand the different stages involved. By looking at each stage, you can see how search engines find and organize web content, impacting how easily your site is found online. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading ## What is Search Engine Crawling? /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Web crawling, as defined by [Google](https://developers.google.com/search/docs/fundamentals/how-search-works#:~:text=Crawling%3A%20Google%20downloads%20text%2C%20images,which%20is%20a%20large%20database.), involves discovering and downloading text, images, and videos from online web pages using search engine web crawlers, also known as spiders. Web crawling ensures that these bots efficiently index content for search engine visibility." /wp:paragraph wp:paragraph These crawlers find new pages by travelling through URLs (links). They crawl sitemaps, [internal links](https://www.lawrencehitches.com/internal-linking-seo/) and backlinks to find additional pages that haven’t been crawled. Once they find a new page, they extract the information to index it in their database. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Different Search Engine Bots & User Agent Strings /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Search engine bots are crucial to understand search engine crawling, and recognizing user agent strings related to those search engine bots enhances this knowledge. /wp:paragraph wp:paragraph Web crawlers, or bots, are automated tools that find new web pages across the internet. By identifying themselves with user-agent strings, they help index online content for better search engine results. /wp:paragraph wp:paragraph {"className":""} Blocking access means no rank on search engines! /wp:paragraph wp:paragraph {"className":""} /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:paragraph {"className":""} /wp:paragraph wp:table | Search Engine | Bot Name | User Agent String Example | Purpose | |---|---|---|---| | Google | Googlebot | Googlebot/2.1 (+http://www.google.com/bot.html) | Googlebot crawls and indexes web pages to help improve Google Search results and other Google services. | | Bing | Bingbot | Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) | Bingbot explores and indexes web pages to enhance Bing Search results. | | DuckDuckGo | DuckDuckBot | DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) | DuckDuckBot scans and indexes web pages to make DuckDuckGo Search better. | | Yandex | YandexBot | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | YandexBot crawls and indexes web pages to improve Yandex Search. | | Baidu | Baiduspider | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | Baiduspider searches and indexes web pages to enhance Baidu Search. | | Apple | Applebot | Applebot/1.0 (+http://www.apple.com/go/applebot) | Applebot explores and indexes web pages for services like Spotlight Search. | /wp:table wp:spacer {"height":"10px"} /wp:spacer wp:heading ## The Stages of Search Engine Crawling /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph [Search engines work](https://www.lawrencehitches.com/how-search-engines-work/) through a process called crawling to deliver relevant search results. This involves five steps: discovery, prioritization, crawling, data extraction, and indexing. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Step One: Discovery /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph The URL discovery process kicks off web crawling. When starting out, search engines like Google use bots, like Googlebot, to track down new, uncrawled pages. /wp:paragraph wp:paragraph Optimizing crawlability is crucial because search engines typically have a huge queue of URLs waiting to be crawled. Ensuring your page gets crawled the first time highlights the importance of optimizing crawlability. /wp:paragraph wp:paragraph Search engine crawling involves several methods to discover new content. /wp:paragraph wp:paragraph Primarily, search engines might re-crawl previously indexed URLs for updates or changes. Additionally, they crawl a website's XML sitemap to uncover new pages. /wp:paragraph wp:paragraph Another approach search engines utilize involves crawling internal and external links to identify new pages. /wp:paragraph wp:paragraph Remember, however, *“Crawling is not a guarantee you’re indexed.” -*[Rand Fishkin](https://sparktoro.com/team/rand) /wp:paragraph wp:paragraph [Search engine indexing](https://www.lawrencehitches.com/search-engine-indexing/) ensures your web pages are added to search results, but it’s not automatic. Your pages must meet specific criteria for search engines to index them. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Step Two: Fetching /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Search engine crawlers kick off by picking a URL from their list. They then request the page from the web server, grabbing and indexing its content to boost search engine results. /wp:paragraph wp:paragraph HTML code is sent by the server as the page’s content, usually in HTML format. This HTML code includes the page's structure and text. Additionally, HTML code may contain links to other resources like images, CSS, stylesheets, and JavaScript files. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Step Three: Parsing /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Web crawling involves fetching the page, where the crawler then parses the HTML content to extract information, a crucial step in web crawling. /wp:paragraph wp:paragraph {"className":""} The information extracted includes the following: /wp:paragraph wp:list - **Links** are identified within the HTML code, including both internal and backlinks. These links are then added to the discover queue for future crawling. - Web crawlers extract **resources** from embedded HTML resources, including images, CSS stylesheets, and JavaScript files. After extraction, web crawlers fetch and analyze these resources separately. - Metadata, such as a page’s title, description, and keywords, is extracted to understand the page’s content and relevance, making metadata crucial for page analysis. /wp:list wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Step 4: Rendering /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Web crawlers struggle with accurately reading content on modern websites built using JavaScript. This makes it tricky for them to interpret and index such sites correctly. /wp:paragraph wp:paragraph To understand a webpage completely, search engines need to render it first. This means they run the JavaScript code on the page, much like how a browser would show it to you. This step is crucial for search engines to grasp the full content and present it correctly. /wp:paragraph wp:paragraph Web crawlers gain a better understanding of web page content, including dynamically generated elements, through this. /wp:paragraph wp:paragraph However, when web crawlers encounter an HTTP error such as HTTP 500, their crawl efficiency may decrease. During these instances, web crawlers typically adjust their crawl rate to prevent server overload. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Step 5: Indexing /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Search engine indexing is when a search engine stores information from web pages in its index. /wp:paragraph wp:paragraph This helps the [search engine's algorithms](https://www.lawrencehitches.com/history-of-search-engines/) decide which pages are most relevant to show when you search for something. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Crawling vs. Indexing /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Crawling and indexing are often confused in SEO, but they are distinct processes. Knowing how each works is key to optimizing your website effectively. /wp:paragraph wp:paragraph Here’s a side-by-side comparison to help: /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:table | Feature | Crawling | Indexing | |---|---|---| | Definition | The process of discovering new or updated web pages | The process of storing and organizing information from web pages | | How it works | Automated bots, known as crawlers, follow hyperlinks to explore web content | Bots analyze the content of crawled pages and extract essential information | | Sources | Hyperlinks, sitemaps, and URL submissions | Pages that have been crawled | | Goal | To collect data from web pages for inclusion in the index | To create a structured and searchable database of web content | | Control | Therobots.txt fileprovides guidance to crawlers on which pages to access | The robots.txt file dictates which pages can be indexed | | Outcome | A comprehensive list of URLs | A searchable database that facilitates efficient retrieval of search results | /wp:table wp:spacer {"height":"10px"} /wp:spacer wp:heading ## How Search Engines Discover and Index Web Pages /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Now that you know more about crawling, let's examine how search engines discover and index web pages. /wp:paragraph wp:paragraph Discovery is the initial phase in how search engines discover and index web pages. Though we’ve mentioned it earlier, a brief overview is essential. During this phase, crawlers need to find pages by: /wp:paragraph wp:list - Bots belonging to search engines crawl the internet via links to discover unknown web pages. - Crawlers investigate sitemaps to locate new pages on a website that haven’t been crawled. - For faster crawling, you can manually submit individual URLs using search engine console tools like Google Search Console." /wp:list wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Indexing /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Understanding how search engines discover and index web pages is key to making your site visible online. Here's how it works: search engines use bots to crawl links and sitemaps, or you can manually submit pages through tools like Google Search Console. /wp:paragraph wp:paragraph During the search engine optimization process, search engines analyze various index factors for SEO efficiency, such as content quality, relevance, and keyword usage. /wp:paragraph wp:paragraph SEO also prioritizes title tags, [meta descriptions](https://www.lawrencehitches.com/how-to-write-meta-descriptions-for-seo/), and header tags (H1, H2, etc.). Internal and external links, image alt text, page load speed, mobile-friendliness, structured data, and social signals play pivotal roles in SEO. /wp:paragraph wp:paragraph Additionally, the freshness of content, domain authority, trustworthiness, user engagement metrics, and robots.txt file demands contribute significantly to SEO. T /wp:paragraph wp:paragraph here are numerous SEO factors determining web pages' indexing efficacy; for a comprehensive list, refer to [Google’s 200 Ranking Factors from Backlinko.](https://backlinko.com/google-ranking-factors) /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading ## FAQ /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### How often do search engine crawlers revisit web pages? /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Search engine crawlers work on their own schedules, so it's hard to predict how often they'll revisit. To get them back more frequently, you can manually submit your URL or sitemap and improve your internal and external links. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### Can search engine crawlers access non-text files like images and videos? /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Yes, they can! While search engine crawlers can't actually view images or videos, they gather information from file names, alt text, captions, and surrounding text to understand the content. This helps them index these files appropriately. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading {"level":3} ### How do search engines discover new pages? /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Search engines find new pages using crawlers. These crawlers track down pages by following links within a site, links from other sites, sitemaps, and sometimes through manual submissions. This way, search engines can efficiently discover new content. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer wp:heading ## Need Help with Your SEO? /wp:heading wp:spacer {"height":"10px"} /wp:spacer wp:paragraph Not sure if your website is optimized for crawling? /wp:paragraph wp:paragraph Reach out for a free [SEO audit](https://www.lawrencehitches.com/seo-audit/) to spot issues and boost your chances of getting indexed and ranked on search engines. /wp:paragraph wp:spacer {"height":"10px"} /wp:spacer Run it with Claude: the same process is packaged as a free [Claude SEO skill](https://www.lawrencehitches.com/claude-seo-skills/) you can drop into Claude Desktop and run as one command. ## Frequently asked questions ### What is the process of registering and indexing a website on a search engine? You do not register a site so much as make it discoverable. Submit a sitemap in Google Search Console and Bing Webmaster Tools, let crawlers fetch your pages, the engine indexes them (stores and understands them), then ranks them for queries. IndexNow can notify Bing of new or changed URLs instantly. --- *Lawrence Hitches is an AI SEO consultant in Melbourne and Chief of Staff at StudioHawk, Australia's most awarded SEO agency. Agent edition of this site: https://www.lawrencehitches.com/md/index.md*