Packages tagged: robots

robotparser
Determine if a useragent can access a URL using robots.txt
Pure Nim library useragent robots robot.txt linux macos win