Post
Yesterday my VPS set off a warning, as it was hit by a huge spike in incoming traffic, peaking at 55GB at 2:15pm and lasting for an hour.
Upon investigating, it turns out it was my PeerTube instance that was targeted.
Where did the traffic come from?
meta-externalagent (aka Meta's web crawler which is used to grab content to train its AI system).
I feel a little bit violated thinking my Fediverse promo video was grabbed by it, sigh.
@_elena Would you be able to use a user agent block list like ai.robots.txt? I have a cron job that updates it daily from their git repo and then restarts nginx.
Except I strip out the part that refers known agents to robots.txt and just give them a 403, because none of them ever honor the robots file anyway.
@EdCates I need to investigate and see if something like this would mess with my YunoHost services... it is not recommended to tweak things on the outside. See a discussion about Anubis: https://forum.yunohost.org/t/is-there-a-way-to-install-anubis-on-yh/40563
@_elena I've been considering setting up a PeerTube site for my personal videos. Is there any defense against Ai bots doing a DDOS? Can they be pre-perma-banned?
@ScottStarkey great question Scott. I have to do some research about this... too late for me now, as Meta went on a pillaging mission with my instance earlier this week. I wonder what @Chocobozzz and the @Framasoft team recommend?
Can't understand much of this thread, but get the gist. Seems like the rebel alliance at work. You guys are wonderful!
@_elena that’s frustrating — especially when it spikes traffic like that without warning.
I’m a Linux/Windows system administrator, and this kind of load can be managed. You can limit or block such crawlers and also protect your VPS with anti-DDoS, rate limiting, and traffic filtering.
If you want, I can help you secure and optimize your setup — or we can provide a VPS with built-in protection.
@_elena They’re doing that on purpose. My hosting provider has already contacted me to say that my site (SearxNG) is causing major traffic issues. Because of this, many small instances may have to be taken offline again. It’s like a digital war...
@_elena Ew. Gross. I feel icky and violated just reading this.
@laserdinosquid thank you for the solidarity 🙏
@_elena@mastodon.social Maybe Meta's AI bots might finally start giving people good advice.
@_elena ugh. That’s just so aggravating. I have read several people mention that the meta bot is being aggressive and crashing sites.
That they can so blatantly steal data is just…
Really hope that the eu is going to do something about their theft.
@_elena not sure if you've seen this https://bluetoot.hardill.me.uk/@ben/116243885816341998, I particularly like his response of using a 301 redirect to a massive file!
@RichBartlett ouf I got a 503 but bookmarked the link. Thanks Rich!
@_elena: due to that I have blocked the whole ASN of Meta Platforms on my server.
Minority of people have complained to me because ”they believed I am hosting malware” (which is far from truth), when they received a link to one of my websites via e.g. Facebook Messenger.
But if there's a risk of sustained violations not only against me, but also others, I have decided to act at least on my side. Do you want the NGINX-compatible blocklist?
@oaj thank you!
my VPS has YunoHost on it and I've been reading documentation about Anubis and the like... it's not recommended to tweak the NGINX config because it may mess up my YH apps.
I just have to live with it I guess... unless I can set something up in Fail2Ban but I'd need a step-by-step guide
@oaj@mastodon.com.pl @_elena@mastodon.social you can have a look at crowdsec too, as an alternative to fail2ban. Their doc is good as far as I remember but everything requires cli to setup.
@_elena
I can imagine that is a terrible feeling.
dang I hope I didn't trigger anything by sharing your video on Facebook. I'm just trying to get some friends and family to come to the fediverse and hopefully delete Facebook (again).
@_elena At least it's trained with high quality data and something useful.
With CC BY-NC-SA license we can't forbid this anyway, but theoretically they aren't allowed to use it for commercial products later on.
@streetcoder yes... and I know that everything posted on the internet is technically fair game but Meta creeps me out so damn much. Thinking that my voice and image are used to train their system is making me feel really violated
@_elena Did you already hear of Anubis to prevent AI scraping? Or maybe you already have it in place?
I think a lot of the Fediverse people are using this to prevent bots: https://github.com/TecharoHQ/anubis
@streetcoder apparently you can't install anubis on top of a YunoHost installation... and even if you did, you'd have to re-install it / tweak it at every YH upgrade, sigh
@_elena That would mean to introduce a whole new workflow on updates or changing the complete tool stack to maybe some Docker based stuff. That sounds a little bit intimidating.
I don't know, we have to find some more experienced admins on this topic.
@streetcoder yes I've heard of it but haven't implemented it yet because I lack the tech skills 🥲
@_elena Sadly I can't I help you on this topic but I'm sure if you post and ask for help someone will step in.
@_elena I was forced to take down my SearXNG instance because of these stupid bots.
@andypiper @ml @_elena my searxng has been fine so far (at least to my knowledge), but thanks for the heads up, i should really put it behind my sso!
@ml 😭 that's awful, I'm sorry
It's ridiculous to think how Meta could DDOS my instance with its stupid bots... thank goodness PeerTube has P2P built in and my video has also been mirrored on many instances, ouf
This is a bonfire demo instance for testing purposes. This is not a production site. There are no backups for now. Data, including profiles may be wiped without notice. No service or other guarantees expressed or implied.
