What is Spider?
A spider (also known as a ‘crawler’) is an automated program that travels through the internet, fetching pages, following links, and compiling an index for search engines. Each crawler marks its path with a user-agent string such as Googlebot or Bingbot. The use of proper site organization along with XML sitemaps and robots.txt instructions helps spiders locate the indexed content while excluding private pages. Knowledge of crawler behaviors assists in guaranteeing comprehensive indexing and that appropriate crawl budgets are being utilized.
Examples include Googlebot indexing new posts, Bingbot updating cached pages, and the audit tool’s spider checking for broken links.
Related terms: crawler, bot, index, user-agent