Protego parses directives set in robots.txt, including non-standard ones such as Request-Rate and Host. It also handles some typos in the directive names.