PieFed 1.7.9 includes Anubis-like scraper blocking
(midwest.social)
Rimu to 
PieFed Meta
AI scrapers have been hammering fediverse servers for years with things getting especially intense lately. Fediverse developers have been slow to respond with solutions so desperate instance admins have been bolting on Anubis which is fragile and really hard to configure in a way that doesn't break federation, mobile clients or web UI access. Anubis forces browsers to run some CPU-heavy javascript for a few seconds, increasing the cost to scrapers hopefully to the point where they go elsewhere instead.
PieFed now includes that functionality as a core feature, not a separate container, proxy or process that needs to be set up. Just set the ANOOBIS environment variable and you're done. Yes, that spelling is intentional.
The amount of work a client needs to do can be tuned with the ANOOBIS_DIFFICULTY_DESKTOP and ANOOBIS_DIFFICULTY_MOBILE environment variables - the defaults are quite weak.
Early results are very promising and a 50% CPU and network reduction on the server is commonly seen. Clients that are not logged in will see a brief loading screen, once every 1000 days, before being taken to the post they were trying to view.
I'm sure this isn't the end of this war but I've been enjoying a respite from it for the last week, long may it continue.
(Image supplied by @dosdude@retrofed.com, thank you!)
If it can be detected reliably, can we use that detection to feed it poisened data. I read it does not take much poisen to invalidate large sets of training data?
I run iocane on my site to poison bots and yeah, scrapers just constantly grab it, to the point where I've exceeded my free vps bandwidth, so it's not free. Granted, I don't mind paying an extra dollar or two for the bandwidth, but my site is also extremely low use so...
I saw that someone on Mastodon tried that. It doesn't stop them they just scrape more and more until your server melts.
Yeah, its not a solution to protect the server, but if enough people do it, it can start having a collective effect.
Its difficult for small operations.
How timely, I just saw piefed.zip/lemmy.zip admins talking about this struggle
What a name for an instance
Just in case you think it's about something dirty, it's actually just about the practice of shaving your hair, beard or otherwise, with water or other lubricant instead of doing it dry.
Okay, that doesn’t make it less a weirdo choice for an instance name. Perhaps even weirder.
It's home of the c/wetshaving comm, which the admin mods. Personally I think it's nice to see the niche comms slowly showing up on the fediverse.
Let's continue our discussion in the chat.
Yeah, my bad. Don't know why I posted here!
Wow this is great!
This is great. Thanks for your work.
Is there anyway this work can be easily adopted for other fediverse software? I know Writefreely is having a huge issue with bots (not necessarily scraping, though maybe that too), but I figure they could use all the help they can get.
We can all use all the help we can get ;)
How does this affect "legitimate" scrapers such as search engines?
I've added exceptions for them.
Wow what a difference
Logged out works exactly as advertised! Can't wait to see the graphs!
I've enabled it and noticed my private instance loading much quicker, so that's neat.
