• Eager Eagle
    link
    English
    122 years ago

    Or don’t do anything. There are plenty of crawlers out there and disallowing won’t stop the unethical ones.

    • @[email protected]
      link
      fedilink
      English
      222 years ago

      Just because some people might break into my house doesn’t mean I’ll stop locking my doors.

      • Eager Eagle
        link
        English
        62 years ago

        that doesn’t lock anything, it’s not a security feature.

        • Arakwar
          link
          fedilink
          42 years ago

          A house door lock isn’t that much about security either.

          • @[email protected]
            link
            fedilink
            English
            42 years ago

            It’s a deterrent. Which is a pretty apt comparison for robots.txt and user agent blocking.

        • @5BC2E7
          link
          02 years ago

          That is the point. They don’t need to be secure to work as a deterrent