Crawling is the process used by search engine web crawlers to visit and download a page and extract its links in order to discover additional pages. Search Engine Journal contributor Jes Scholz has published an article explaining website crawling. She says, “A web crawler works by discovering URLs and downloading the page content. During this process, they may pass the content over to the search engine index and will extract links to other web pages. These found links will fall into different categorizations: New URLs that are unknown to the search engine. Known URLs that give no guidance... [...]