• drphungky
    link
    English
    33 months ago

    Anyone using bots that are openly bots would be limiting their shit to robots.txt. Anyone who is out trawling the whole website is probably obfuscating what they’re doing by spoofing new user agents.