Just deValk says, “Search engines crawl your site to get the contents into their index. The bigger your site gets, the longer this crawl takes. It’s important that the time spent crawling your site is well spent. If your site has a 1,000 pages or less, this is not a topic you’ll need to think about much. If  you intend on growing your site though, keep on reading. Acquiring some good habits early on can safe you from huge headaches later on. In this article we’ll cover what crawl efficiency is and what you can do about it.

All search engines crawl the same way. In this article we’ll refer to Google and Googlebot.

How does a crawl of your site work?

Google finds a link to your site somewhere on the web. At that point that URL is the beginning of a virtual pile. The process is pretty easy after that:

  1. Googlebot takes one page from that pile;
  2. it crawls the page and indexes all the contents for use in Google;
  3. it then adds all the links on that page to the pile”.

Crawl efficiency: making Google’s crawl easier

Yoast Blog

Sharing is caring