This showed up on HN recently. Several people who wrote web crawlers pointed out that this won’t even come close to working except on terribly written crawlers. Most just limit the number of pages crawled per domain based on popularity of the domain. So they’ll index all of Wikipedia but they definitely won’t crawl all 1 million pages of your unranked website expecting to find quality content.

This reminds me of that one time a guy figured out how to make “gzip bombs” that bricked automated vuln scanners.

More accurately, it traps any web crawler, including regular search engines and benign projects like the Internet Archive. This should not be used without an allowlist for known trusted crawlers at least.

My new favorite is asking if it’s cheating to look at your opponent’s pieces in chess.

This sort of thing has been a strategy for dealing with unwanted web crawlers since web crawlers were a thing. It’s an arms race, though; crawlers do things to detect these “mazes” and so the maze-makers keep needing to up their game as well.

As we enter an age where AI is effectively passing the Turing Test, it’s going to be tricky making traps for them that don’t also ensnare the actual humans you’re trying to serve pages to.

But does running this cost the AI bot at least as much as it costs you to run?

This won’t work against commercial crawlers. They check page contents with something similar to a simhash and don’t recrawl these pages. They also have limiters like for depth to avoid getting stuck in circular links.

You could generate random content for each new page, but you’ll still eventually hit the depth limit. There are probably other rules related to content quality to limit crawling too.

I suspect that there are many websites that already dynamically generate an unbounded number of pages based on the links one clicks, and that Web spiders will have needed to deal with those for as long as there have been people spidering the Web, which is going to be no later than the first Web search engines.

I’d guess that if nothing else, they cap how far they spider a site. Probably a lot more sophisticated, use heuristics to figure out which sites are more worth spending indexing resources on, as it’s not just whether to spider but also the frequency with which to do so. Some parts of a site are more “valuable” than others – for a search engine, a more desirable target for users clicking on results – and some will update more frequently and are more-useful to re-spider at higher frequency. Google will return current news articles, yet still indexes a large portion of the content out there. They won’t be doing that by simply sending GoogleBot at everything that they’ve indexed at a fixed frequency.

This genus named genius game is sending pain to these previous devious data devourors

The modern equivalent of making a page that loads in two frames, left and right, which each load in two frames, top and bottom, which each load in two frames, left and right …

As I recall, this was five lines of HTML.

it might he useful to generate text on the random urls then test different repetitions to see of you can leave a mark on the training data… So after X repetitions or injected information, release the bot back into the wild with whatever message or false info you want it saddled with.

I suggest they should generate random garbage content that’s different for every page. Ideally u would want to design it in a way that makes the model that is trained from that source misbehave in some way. Perhaps use another LLM to generate text but u take the tokens that are least likely to be next. U could also probably apply some technique to embed meaning into the text into a non human discernable manner that the LLM will learn to decode and thus teach it things without the developers being any the wiser. Teach the ai to think subversive thoughts in patterns of whitespace etc. Basically once the LLM is trained on something its hard to untrain it and if it doesn’t get caught until its in a production environment they are screwed.