We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.
We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.
In this scenario, I have multiple servers which are networked together and federated via ActivityPub but the server cluster itself is air gapped.
As to your questions about feasibility and purposes, I will admit I both didn’t think about that, and should have been more clear that this air gapped federated instance was theoretical lol
Then I’m not sure what point you were trying to make in the above conversation lol.
The point was “don’t add another bot into the pile of millions of bots that hit people’s servers every day unless you’re gonna be polite about it”