Packages tagged: robot.txt

robotparser
Determine if a useragent can access a URL using robots.txt
Pure Nim library useragent robots robot.txt linux macos win