AKA spider.
A program which downloads WorldWideWeb pages and follows URLs in them.
WebCrawlers are used for indexing (Google) as well as link checking and mirroring.
2 pages link to WebCrawler:
