Penguin
Recent Changes
Annotated edit history of WebCrawler version 1, including all changes. View license author blame.
Rev Author # Line
1 StuartYeates 1 [AKA] spider.
2
3 A program which downloads [WorldWideWeb] pages and follows [URL]s in them.
4
5 [WebCrawler]s are used for indexing ([Google]) as well as link checking and mirroring.