cross-posted from: https://lemmy.sdf.org/post/58474758

Someone brought the HTtrack app to my attention in this thread. Superficially it’s a great concept. But it did not work for me.

The author of HTtrack says they respect the robots.txt files. I’m not sure if this is my problem. But what an absurd line to draw. Why should a realtime GUI user have more privilege to view a website than a CLI user?

If anything, it should be the other way around.

  • People who lack the priviledge of having Internet at home face this nasty discrimination of being treated like a bot after they make the extra effort of commuting to a public library just to fetch a website for offline viewing later. This 2nd-classing of a demographic who is already marginalized is quite despicable.
  • People with Internet at home can schedule HTtrack to run at an off-peak time with a narrow bandwidth put less burden on the server than the realtime GUI users who obviously hit the site mostly during peak times.
  • (update) Some people are on measured rate Internet connections that give tiny daytime quotas and generous late night quotas. Tools like HTtrack are needed to manage this. Which ultimately benefits the more privileged Internet users who have no constraints.

Update

There is a poorly worded -s0 option to ignore robots.txt. Fooled some people into thinking the tool uses robots.txt to direct the fetches.

  • historicaldocuments
    2·
    17 days ago

    It’s important to note that there’s no difference at all from the server’s side, other than an implication of how often scraping happens or how the data is being used.

    I thought about maybe checking server logs for what files a scraper might skip just for bandwidth or whatever, but I don’t know much about that world.

    Which is why the GUI restriction was added to robots.txt in the first place.

    I had no idea there was a GUI restriction since I stopped digging pretty quickly about it. Upon further reading it’s apparently been used in several lawsuits successfully as a sort of “no tresspassing” sign with some court precedent to it. Neat.