ruby-webrobots

Library for creating robots.txt-aware web robots
  https://github.com/knu/webrobots
  0
  no reviews


This library helps write robots.txt-compliant web robots in Ruby, based on Nokogiri's functionality.