Welcome to Incremental Social! Learn more about this project here!
Check out lemmyverse to find more communities to join from here!

corroded ,

Speaking from experience, be careful you don't become over-zealous in your anti-scraping efforts.

I often buy parts and equipment from a particular online supplier. I also use custom inventory software to catalog my parts. In the past, I could use cURL to pull from their website, and my software would parse the website and save part specifications to my local database.

They have since enacted intense anti-scraping measures, to the point that cURL no longer works. I've had to resort to having the software launch Firefox to load the web page, then the software extracts the HTML from Firefox.

I doubt that their goal was to block customers from accessing data for items they purchased, but that's exactly what they did in my case. I've bought thousands of dollars of product from them in the past, but this was enough of a pain in the ass to make me consider switching to a supplier with a decent API or at least a less restrictive website.

Simple rate limiting may have been a better choice.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • privacy@lemmy.ml
  • random
  • incremental_games
  • meta
  • All magazines