Crawler

A Crawler is a sophisticated computer software program that searches the Internet for select information. Search engines use Crawlers to search web pages for keywords and other information, so they can point you to sites you that are most like what you're looking for.

Crawlers (also called a Web Robot or a Spider) will download a copy of the HTML content (not including graphics) from web sites. The copies are stored and later processed and indexed by the search engine to provide fast searches and up-to-date data.

Crawlers are the empowering force behind search engines. Without Crawlers, all search engines (such as Google and Yahoo) could not function. Instead of the Internet bringing you the world at your fingertips, a world without Crawlers would mean we'd need to physically search through Internet "phone books" ourselves, category by category, and city by city.

Not all Crawlers are good, though. They sure make navigating the web easy and effortless, but Crawlers are the main source of trouble out there, too. Crawler Viruses, Worms, and Spambots are some negative examples of Crawlers. CAPTCHA is a popular security feature that keeps most Crawlers from damaging websites or gathering information without permission (such as collecting email addresses and then spamming them.)

Crawlers go by many names: they're also called Bots, Ants, Worms, Web Scutters, Automatic Indexers, Spiders, Web spider, and Web robot.