Spider handles the robots.txt, scraping, collecting and looping so that you can just handle the data.

spider is useful to applications that need gather data from web sites.