Welcome to Incremental Social! Learn more about this project here!
Check out lemmyverse to find more communities to join from here!

Is there a simple way to severly impede webscraping and LLM data collection of my website?

I am working on a simple static website that gives visitors basic information about myself and the work I do. I want this as a way use to introduce myself to potential clients, collaborators, etc., rather than rely solely on LinkedIn as my visiting card.

This may seem sound rather oxymoronic given that I am literally going to be placing (some relevant) details about myself and my work on the internet, but I want to limit the websites' access from bots, web scraping and content collection for LLMs.

Is the a realistic expectation?

Also, any suggestions on privacy respecting, yet inexpensive domains that I can purchase in Europe would be of super great help.

habitualTartare ,

https://en.wikipedia.org/wiki/Robots.txt

Should cover any polite web crawlers but it is voluntary.

https://platform.openai.com/docs/gptbot

Might have to put it behind a captcha or other type to severely limit automated access.

It's not realistic to assume it won't get scraped eventually. Such as someone paying people to bypass capatcha or web crawlers that don't respect robots.txt. I also don't know if Google and Microsoft bundle their AI data collection that doesn't also remove your site from web search.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • privacy@lemmy.ml
  • random
  • incremental_games
  • meta
  • All magazines
    •