OpenAI's crawler just found our family server / cloud services and immediately proceeded to crash Nextcloud within minutes. Fucking fantastic.
Is there some nice, up-to-date write-up on the different tools to protect yourself against this?
#AI #AISlop #AttackOfTheMachines #selfHosting
@lunareclipse @kura Ours are also login-walled, but OpenAI crashed our worker threads nevertheless somehow.
Thanks for the link! 
@nelle @lunareclipse Out of curiosity: Would it also poison "respecting" crawlers that read the robots.txt? 🤔
I noticed an *immense* difference in company behaviour between the US shitheads (which all behave maliciously), data brokers (also highly maliciously) and e.g. EU companies who seem to actually abort if instructed.
In general I'm more than happy to serve some malicious compliance to those dickheads, our server should have the resources for that.
@Natanox It's a bit of work, but I'd suggest something like #NetBird or #tailscale to keep your private things private.
The only real downside I see so far is that on mobile devices (iOS in my case) it increases battery consumption to a noticeable degree.
@Fishd Not all family members are inclined to install these tools everywhere, and it would cause e.g. Nextcloud password-protected Share Links to stop working for anyone we want to send things to.
I'll probably go with something like Anubis + iocane instead.
@Natanox Fair points.
My concern with those tools are, you're just playing whack-a-mole ... and your opponent has more resources than you and is sufficiently motivated (by the way of investment capital) to defeat you.
Similarly for those folks suggesting fighting back by 'poisoning the well' ... that assumes you've the spare compute power and significant energy supplies.
@Fishd Also fair points.
Though these tools that fight back are a community effort, so there's a lot of brainpower going in there as well. Basically the giant well-funded army with only a few bright minds against a motivated guerilla force, but digital.
It's a shitty situation, but for now I want to keep our infrastructure as accessible as possible to convince some family members that corpo clouds indeed aren't inherently better.
@FoxVK @Fishd If AI crawlers found us I'll rather directly go with things like Anubis. Even *if* some of them care enough, a sufficient amount of them do not or even actively circumvent blocks (e.g. Anthropic is known for that).
To be fair though, I didn't check the default robots.txt if the Nextcloud docker before so this might've been preventable. But at least now I know the bad actors will get to us soon.
@nigelharpur I learned today that the robots.txt that comes with a default Nextcloud install (at least in a docker) is perfectly basic. I guess a basic measure would be to replace that one with a robots.txt that includes all the anti-AI stuff telling OpenAI and others to screw off.
Other than that most answers mentioned Anubis and its alternatives as well as poisoning tools like iocaine.
@Natanox I can't recommend, but will someone please comment with things to look out for?
Obviously, I can look for average traffic off the chart, but does the creepy-crawler traffic come from one IP address, or a narrow range? Does it deliberately ignore robots.txt ? These things could trigger a goooo_sloooow, or return a "Go stick your head in a pig" page, or something.
@Natanox here are the resources I promised:
Database of IP reputation: https://nerd.cesnet.cz/
Lots of useful categorized IP lists: https://iplists.firehol.org/
Natanox 🇺🇦🇵🇸
Kura 
:kura_explode:
luna the doggie 
Matthias Bach
"Musty Bits" McGee
nelle
Fishd
FoxVK
Nigel
Arcane Roops
Lost Librarian