Cynicus Rex to [email protected]English • 1 month agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square63fedilinkarrow-up1104arrow-down133
arrow-up171arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex to [email protected]English • 1 month agomessage-square63fedilink
minus-squareDa Bald Eagullinkfedilink35•1 month agoThat is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
minus-square@[email protected]linkfedilink11•1 month agoLie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
minus-squareJackbyDevlinkfedilinkEnglish-4•1 month agoIt’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-square@[email protected]linkfedilink3•1 month agoMaybe? But it’s not like that’s the only alternative thing to say, lol
minus-squaremoxlinkfedilink4•1 month agoAssuring someone that they have control of something and the safety that comes with it, when in fact they do not, is well outside the realm of a simplification. It’s just plain false. It can even be dangerous.
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
Is it a lie or a simplification for beginners?
Lie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Maybe? But it’s not like that’s the only alternative thing to say, lol
Assuring someone that they have control of something and the safety that comes with it, when in fact they do not, is well outside the realm of a simplification. It’s just plain false. It can even be dangerous.
the word disallow is right there