I am working on a simple static website that gives visitors basic information about myself and the work I do. I want this as a way use to introduce myself to potential clients, collaborators, etc., rather than rely solely on LinkedIn as my visiting card.
This may seem sound rather oxymoronic given that I am literally going to be placing (some relevant) details about myself and my work on the internet, but I want to limit the websites’ access from bots, web scraping and content collection for LLMs.
Is this a realistic expectation?
Also, any suggestions on privacy respecting, yet inexpensive domains that I can purchase in Europe would be of super great help.
@Maroon fwiw I just added a robots txt with a lot of Gen AI user agents disallowed. Ive added this to Google search console to see if I have any way of checking if the AI companies honour robots txt. I used the following link to get the robots.txt template: https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/
Yeah like anyone is obeying robots.txt…lmfao
@Dkarma well exactly. I don’t expect most of them to but if bot paths are tracked in search console (no idea if they are) I might be able to see them