@zephyr to Lemmy [email protected] • 2 years agoWill Lemmy posts be indexed by search engines?message-square10arrow-up121arrow-down10
arrow-up121arrow-down1message-squareWill Lemmy posts be indexed by search engines?@zephyr to Lemmy [email protected] • 2 years agomessage-square10
minus-squareMattlinkfedilinkEnglish2•2 years agoI checked my instance, and here’s the contents of the robots.txt file. User-Agent: * Disallow: /login Disallow: /settings Disallow: /create_community Disallow: /create_post Disallow: /create_private_message Disallow: /inbox Disallow: /setup Disallow: /admin Disallow: /password_change Disallow: /search/ Legitimate search engines will index everything, except what’s disallowed. Of course, the robots.txt could be changed to block all indexing by legitimate search engines.
I checked my instance, and here’s the contents of the
robots.txtfile.User-Agent: * Disallow: /login Disallow: /settings Disallow: /create_community Disallow: /create_post Disallow: /create_private_message Disallow: /inbox Disallow: /setup Disallow: /admin Disallow: /password_change Disallow: /search/Legitimate search engines will index everything, except what’s disallowed. Of course, the
robots.txtcould be changed to block all indexing by legitimate search engines.