Das Thema AI-Crawler, Bots und Scraper macht auch vor dem Fediverse nicht halt. Nicht nur, dass hier unerwünscht Daten von Usern "abgesaugt" werden, diese Abfragen erzeugen auch immer mehr Last auf den Servern - viel unnötiger Traffic, der da entsteht und Ressourcen kostet.

Vieles lässt sich durch die robots.txt aussperren, aber leider hält sich nicht jeder Bot an diese. Natürlich ist auch von hartnäckigen Bots und Spidern ein Ausschluss in der .htaccess oder Apache-Config möglich. Auch mit Fail2Ban lässt sich da eine Menge bauen.

Allerdings ist es auch empfehlenswerter, diese IPs gleich kernelnah, also z.B. über nftables, zu sperren. Leider sind es nicht nur ein paar IPs, leider kann man nicht alle IPs dieser Bots sperren, weil auch immer wieder neue IPs hinzukommen. Es gibt aber bestimmte IPs und Adressbereiche, die von diesen Bots genutzt werden. Die Infos habe ich bei den "Anbietern" selbst gefunden, mit Ausnahme von SEMrush, der wirklich sehr aggressiv crawlen kann und gerne mal die robots.txt völlig ignoriert:

openai.com/gptbot.json
openai.com/chatgpt-user.json
openai.com/searchbot.json
docs.claude.com/en/api/ip-addr…
developers.google.com/static/s…
bing.com/toolbox/bingbot.json
search.developer.apple.com/app…
perplexity.com/perplexitybot.j…
perplexity.com/perplexity-user…
duckduckgo.com/duckduckgo-help…
index.commoncrawl.org/ccbot.js…
ipinfo.io/AS209366

Ja, das sind natürlich nicht alle Bots, Crawler und Co! Das ist mir bewusst. Aber ein großer Teil Traffic verschwindet damit schon!

Da ich auf meinen Friendica-Instanzen aber auch keinen Besuch von Google und Co. haben möchte, habe ich heute mal einen Rundumschlag gemacht und diese IPs von oben genannten Seiten in nftables reingeworfen. Ergebnis: Es ist bedeutend ruhiger geworden! Und dabei sind es bei weitem natürlich nicht alle Bots und Crawler, aber doch schon einige!

Ich weiß nicht, ob jemand Interesse an den IPs hat, ich stelle diese hier einfach mal zur Verfügung. Quelle sind wie bereits erwähnt oben stehende Seiten. Nutzung natürlich auf eigene Gefahr. 😉

IPv4-Adressen:

132.196.86.0/24,172.182.202.0/25,172.182.204.0/24,172.182.207.0/25,172.182.214.0/24,172.182.215.0/24,20.125.66.80/28,20.171.206.0/24,20.171.207.0/24,4.227.36.0/25,52.230.152.0/24,74.7.175.128/25,74.7.227.0/25,74.7.227.128/25,74.7.228.0/25,74.7.230.0/25,74.7.241.0/25,74.7.241.128/25,74.7.242.0/25,74.7.243.128/25,74.7.244.0/25,104.210.139.192/28,104.210.139.224/28,13.65.138.112/28,13.65.138.96/28,13.67.46.240/28,13.67.72.16/28,13.70.107.160/28,13.71.2.208/28,13.76.115.224/28,13.76.115.240/28,13.76.116.80/28,13.76.223.48/28,13.76.32.208/28,13.79.43.0/28,13.83.167.128/28,13.83.237.176/28,132.196.82.48/28,135.119.134.128/28,135.119.134.192/28,135.220.73.208/28,135.220.73.240/28,135.237.131.208/28,135.237.133.48/28,137.135.183.96/28,137.135.190.240/28,137.135.191.176/28,137.135.191.32/28,138.91.30.48/28,138.91.46.96/28,168.63.252.240/28,172.178.140.144/28,172.178.141.112/28,172.178.141.128/28,172.183.143.224/28,172.183.222.128/28,172.196.40.208/28,172.202.102.112/28,172.204.16.64/28,172.204.27.16/28,172.212.159.64/28,172.213.11.144/28,172.213.12.112/28,172.213.21.112/28,172.213.21.144/28,172.213.21.16/28,172.215.218.96/28,191.233.1.112/28,191.233.1.128/28,191.233.1.224/28,191.233.194.32/28,191.233.196.112/28,191.233.199.160/28,191.233.2.0/28,191.234.167.128/28,191.235.66.16/28,191.235.98.144/28,191.235.99.80/28,191.237.249.64/28,191.239.245.16/28,20.0.53.96/28,20.102.212.144/28,20.117.22.224/28,20.125.112.224/28,20.125.144.144/28,20.161.75.208/28,20.168.7.192/28,20.168.7.240/28,20.169.72.112/28,20.169.72.96/28,20.169.73.176/28,20.169.73.32/28,20.169.73.64/28,20.169.78.112/28,20.169.78.128/28,20.169.78.144/28,20.169.78.160/28,20.169.78.176/28,20.169.78.192/28,20.169.78.208/28,20.169.78.48/28,20.169.78.64/28,20.169.78.80/28,20.169.78.96/28,20.169.86.224/28,20.169.86.240/28,20.169.87.112/28,20.172.29.32/28,20.193.233.240/28,20.193.50.32/28,20.194.0.208/28,20.194.1.0/28,20.194.157.176/28,20.198.67.96/28,20.203.245.32/28,20.204.24.240/28,20.206.107.192/28,20.210.154.128/28,20.210.174.208/28,20.210.211.192/28,20.215.187.208/28,20.215.188.192/28,20.215.214.16/28,20.215.219.128/28,20.215.219.160/28,20.215.219.208/28,20.215.220.112/28,20.215.220.128/28,20.215.220.144/28,20.215.220.160/28,20.215.220.176/28,20.215.220.192/28,20.215.220.208/28,20.215.220.64/28,20.215.220.80/28,20.215.220.96/28,20.226.32.80/28,20.227.140.32/28,20.228.106.176/28,20.235.75.208/28,20.235.87.224/28,20.249.63.208/28,20.27.94.128/28,20.45.178.144/28,20.55.229.144/28,20.63.221.64/28,20.90.7.144/28,20.97.189.96/28,23.102.140.144/28,23.102.141.32/28,23.97.109.224/28,23.98.142.176/28,23.98.179.16/28,23.98.186.176/28,23.98.186.192/28,23.98.186.64/28,23.98.186.96/28,4.151.119.48/28,4.151.241.240/28,4.151.71.176/28,4.189.118.208/28,4.189.119.48/28,4.196.118.112/28,4.196.198.80/28,4.197.115.112/28,4.197.19.176/28,4.197.22.112/28,4.197.64.0/28,4.197.64.16/28,4.197.64.48/28,4.197.64.64/28,4.205.128.176/28,40.116.73.208/28,40.122.235.112/28,40.67.183.160/28,40.67.183.176/28,40.75.14.224/28,40.81.134.128/28,40.81.134.144/28,40.81.234.144/28,40.84.181.32/28,40.84.221.208/28,40.84.221.224/28,51.107.70.192/28,51.8.155.48/28,51.8.155.64/28,51.8.155.80/28,52.148.129.32/28,52.154.22.48/28,52.156.77.144/28,52.159.227.32/28,52.159.249.96/28,52.165.212.16/28,52.165.212.32/28,52.165.212.48/28,52.172.129.160/28,52.172.251.112/28,52.173.123.0/28,52.173.219.112/28,52.173.219.96/28,52.173.221.16/28,52.173.221.176/28,52.173.221.208/28,52.173.234.16/28,52.173.234.80/28,52.173.235.80/28,52.176.139.176/28,52.187.246.128/28,52.190.137.144/28,52.190.137.16/28,52.190.139.48/28,52.190.142.64/28,52.190.190.16/28,52.225.75.208/28,52.230.163.32/28,52.230.164.176/28,52.231.30.48/28,52.231.34.176/28,52.231.39.144/28,52.231.39.192/28,52.231.49.48/28,52.231.50.64/28,52.236.94.144/28,52.242.132.224/28,52.242.132.240/28,52.242.245.208/28,52.252.113.240/28,52.255.109.112/28,52.255.109.128/28,52.255.109.144/28,52.255.109.80/28,52.255.109.96/28,52.255.111.0/28,52.255.111.112/28,52.255.111.16/28,52.255.111.32/28,52.255.111.48/28,52.255.111.80/28,57.151.131.224/28,57.154.174.112/28,57.154.175.0/28,57.154.187.32/28,68.154.28.96/28,68.218.30.112/28,68.220.57.64/28,68.221.67.160/28,68.221.67.192/28,68.221.67.224/28,68.221.67.240/28,68.221.75.16/28,74.226.253.160/28,74.249.86.176/28,74.7.35.112/28,74.7.35.48/28,74.7.36.64/28,74.7.36.80/28,74.7.36.96/28,9.160.163.224/28,9.160.164.128/28,104.210.140.128/28,135.234.64.0/24,172.182.193.224/28,172.182.193.80/28,172.182.194.144/28,172.182.194.32/28,172.182.195.48/28,172.182.209.208/28,172.182.211.192/28,172.182.213.192/28,172.182.224.0/28,172.203.190.128/28,20.14.99.96/28,20.168.18.32/28,20.169.6.224/28,20.169.7.48/28,20.169.77.0/25,20.171.123.64/28,20.171.53.224/28,20.25.151.224/28,20.42.10.176/28,4.227.36.0/25,40.67.175.0/25,40.90.214.16/28,51.8.102.0/24,74.7.175.128/25,74.7.228.0/25,74.7.228.128/25,74.7.229.0/25,74.7.229.128/25,74.7.230.0/25,74.7.241.128/25,74.7.242.128/25,74.7.243.0/25,74.7.244.0/25,160.79.104.0/21,192.178.4.0/27,192.178.4.128/27,192.178.4.160/27,192.178.4.192/27,192.178.4.224/27,192.178.4.32/27,192.178.4.64/27,192.178.4.96/27,192.178.5.0/27,192.178.6.0/27,192.178.6.128/27,192.178.6.160/27,192.178.6.192/27,192.178.6.224/27,192.178.6.32/27,192.178.6.64/27,192.178.6.96/27,192.178.7.0/27,192.178.7.128/27,192.178.7.160/27,192.178.7.192/27,192.178.7.224/27,192.178.7.32/27,192.178.7.64/27,192.178.7.96/27,34.100.182.96/28,34.101.50.144/28,34.118.254.0/28,34.118.66.0/28,34.126.178.96/28,34.146.150.144/28,34.147.110.144/28,34.151.74.144/28,34.152.50.64/28,34.154.114.144/28,34.155.98.32/28,34.165.18.176/28,34.175.160.64/28,34.176.130.16/28,34.22.85.0/27,34.64.82.64/28,34.65.242.112/28,34.80.50.80/28,34.88.194.0/28,34.89.10.80/28,34.89.198.80/28,34.96.162.48/28,35.247.243.240/28,66.249.64.0/27,66.249.64.128/27,66.249.64.160/27,66.249.64.192/27,66.249.64.224/27,66.249.64.32/27,66.249.64.64/27,66.249.64.96/27,66.249.65.0/27,66.249.65.128/27,66.249.65.160/27,66.249.65.192/27,66.249.65.224/27,66.249.65.32/27,66.249.65.64/27,66.249.65.96/27,66.249.66.0/27,66.249.66.128/27,66.249.66.160/27,66.249.66.192/27,66.249.66.224/27,66.249.66.32/27,66.249.66.64/27,66.249.66.96/27,66.249.67.0/27,66.249.67.32/27,66.249.67.64/27,66.249.68.0/27,66.249.68.128/27,66.249.68.160/27,66.249.68.192/27,66.249.68.32/27,66.249.68.64/27,66.249.68.96/27,66.249.69.0/27,66.249.69.128/27,66.249.69.160/27,66.249.69.192/27,66.249.69.224/27,66.249.69.32/27,66.249.69.64/27,66.249.69.96/27,66.249.70.0/27,66.249.70.128/27,66.249.70.160/27,66.249.70.192/27,66.249.70.224/27,66.249.70.32/27,66.249.70.64/27,66.249.70.96/27,66.249.71.0/27,66.249.71.128/27,66.249.71.160/27,66.249.71.192/27,66.249.71.224/27,66.249.71.32/27,66.249.71.64/27,66.249.71.96/27,66.249.72.0/27,66.249.72.128/27,66.249.72.160/27,66.249.72.192/27,66.249.72.224/27,66.249.72.32/27,66.249.72.64/27,66.249.73.0/27,66.249.73.128/27,66.249.73.160/27,66.249.73.192/27,66.249.73.224/27,66.249.73.32/27,66.249.73.64/27,66.249.73.96/27,66.249.74.0/27,66.249.74.128/27,66.249.74.160/27,66.249.74.192/27,66.249.74.224/27,66.249.74.32/27,66.249.74.64/27,66.249.74.96/27,66.249.75.0/27,66.249.75.128/27,66.249.75.160/27,66.249.75.192/27,66.249.75.224/27,66.249.75.32/27,66.249.75.64/27,66.249.75.96/27,66.249.76.0/27,66.249.76.128/27,66.249.76.160/27,66.249.76.192/27,66.249.76.224/27,66.249.76.32/27,66.249.76.64/27,66.249.76.96/27,66.249.77.0/27,66.249.77.128/27,66.249.77.160/27,66.249.77.192/27,66.249.77.224/27,66.249.77.32/27,66.249.77.64/27,66.249.77.96/27,66.249.78.0/27,66.249.78.128/27,66.249.78.160/27,66.249.78.32/27,66.249.78.64/27,66.249.78.96/27,66.249.79.0/27,66.249.79.128/27,66.249.79.160/27,66.249.79.192/27,66.249.79.224/27,66.249.79.32/27,66.249.79.64/27,157.55.39.0/24,207.46.13.0/24,40.77.167.0/24,13.66.139.0/24,13.66.144.0/24,52.167.144.0/24,13.67.10.16/28,13.69.66.240/28,13.71.172.224/28,139.217.52.0/28,191.233.204.224/28,20.36.108.32/28,20.43.120.16/28,40.79.131.208/28,40.79.186.176/28,52.231.148.0/28,20.79.107.240/28,51.105.67.0/28,20.125.163.80/28,40.77.188.0/22,65.55.210.0/24,199.30.24.0/23,40.77.202.0/24,40.77.139.0/25,20.74.197.0/28,20.15.133.160/27,40.77.177.0/24,40.77.178.0/23,17.241.208.160/27,17.241.193.160/27,17.241.200.160/27,17.22.237.0/24,17.22.245.0/24,17.22.253.0/24,17.241.75.0/24,17.241.219.0/24,17.241.227.0/24,17.246.15.0/24,17.246.19.0/24,17.246.23.0/24,107.20.236.150/32,3.224.62.45/32,18.210.92.235/32,3.222.232.239/32,3.211.124.183/32,3.231.139.107/32,18.97.1.228/30,18.97.9.96/29,44.208.221.197/32,34.193.163.52/32,18.97.21.0/30,18.97.43.80/29,130.107.228.224/32,104.43.54.127/32,104.43.55.116/32,104.43.55.117/32,104.43.55.166/32,104.43.55.167/32,108.141.83.74/32,13.89.106.77/32,172.169.17.165/32,191.233.3.197/32,191.233.3.202/32,191.234.216.178/32,191.234.216.4/32,191.235.201.214/32,191.235.202.38/32,191.235.202.48/32,20.113.14.159/32,20.113.3.121/32,20.12.141.99/32,20.185.79.15/32,20.185.79.47/32,20.191.44.119/32,20.191.44.16/32,20.191.44.22/32,20.191.44.234/32,20.191.45.212/32,20.193.12.126/32,20.193.24.10/32,20.193.24.251/32,20.193.25.197/32,20.193.27.215/32,20.193.45.113/32,20.195.108.47/32,20.197.209.11/32,20.197.209.27/32,20.201.15.208/32,20.204.240.172/32,20.204.241.148/32,20.204.242.101/32,20.204.242.19/32,20.204.243.55/32,20.204.246.254/32,20.204.246.81/32,20.207.107.181/32,20.207.72.11/32,20.207.72.110/32,20.207.72.113/32,20.207.72.21/32,20.207.97.190/32,20.207.99.197/32,20.219.43.246/32,20.219.45.190/32,20.219.45.67/32,20.226.133.105/32,20.232.184.230/32,20.3.1.178/32,20.40.133.240/32,20.43.150.85/32,20.43.150.93/32,20.43.172.120/32,20.44.222.1/32,20.49.136.28/32,20.50.168.91/32,20.50.48.159/32,20.50.48.192/32,20.50.49.0/32,20.50.49.237/32,20.50.49.25/32,20.50.49.40/32,20.50.49.55/32,20.50.50.118/32,20.50.50.121/32,20.50.50.123/32,20.50.50.130/32,20.50.50.134/32,20.50.50.145/32,20.50.50.146/32,20.50.50.163/32,20.50.50.46/32,20.53.134.160/32,20.53.78.106/32,20.53.78.123/32,20.53.78.138/32,20.53.78.144/32,20.53.78.236/32,20.53.91.2/32,20.53.92.211/32,20.56.197.58/32,20.56.197.63/32,20.61.34.40/32,20.62.224.44/32,20.71.12.143/32,20.72.242.93/32,20.73.132.240/32,20.73.202.147/32,20.75.144.152/32,20.79.226.26/32,20.79.238.198/32,20.79.239.66/32,20.80.129.80/32,20.93.28.24/32,20.99.255.235/32,4.156.136.79/32,4.182.131.108/32,4.195.133.120/32,4.209.224.56/32,4.213.46.14/32,4.228.76.163/32,40.114.182.153/32,40.114.182.172/32,40.114.182.45/32,40.114.183.196/32,40.114.183.251/32,40.114.183.88/32,40.119.232.146/32,40.119.232.215/32,40.119.232.218/32,40.119.232.251/32,40.119.232.50/32,40.64.105.247/32,40.64.106.11/32,40.76.162.191/32,40.76.162.208/32,40.76.162.247/32,40.76.163.23/32,40.76.163.7/32,40.76.173.151/32,40.80.242.63/32,40.81.250.205/32,40.88.21.235/32,40.89.243.175/32,51.104.146.225/32,51.104.146.235/32,51.104.160.167/32,51.104.160.177/32,51.104.161.32/32,51.104.162.149/32,51.104.163.250/32,51.104.164.109/32,51.104.164.147/32,51.104.164.189/32,51.104.164.215/32,51.104.166.111/32,51.104.167.104/32,51.104.167.110/32,51.104.167.19/32,51.104.167.52/32,51.104.167.54/32,51.104.167.61/32,51.104.167.71/32,51.104.167.87/32,51.104.167.88/32,51.104.167.95/32,51.104.167.96/32,51.104.180.26/32,51.104.180.47/32,51.104.180.53/32,51.107.40.209/32,51.116.131.221/32,51.120.48.122/32,51.138.90.161/32,51.138.90.206/32,51.138.90.233/32,51.8.253.152/32,51.8.71.117/32,52.142.24.149/32,52.142.26.175/32,52.143.241.111/32,52.143.242.6/32,52.143.243.117/32,52.143.244.81/32,52.143.247.235/32,52.143.95.162/32,52.143.95.204/32,52.146.58.236/32,52.146.59.12/32,52.146.59.154/32,52.146.59.156/32,52.146.63.80/32,52.148.161.87/32,52.148.165.38/32,52.149.25.43/32,52.149.28.18/32,52.149.28.83/32,52.149.30.45/32,52.149.56.151/32,52.149.58.139/32,52.149.58.173/32,52.149.58.27/32,52.149.58.69/32,52.149.60.38/32,52.149.61.51/32,52.154.169.200/32,52.154.169.50/32,52.154.170.113/32,52.154.170.117/32,52.154.170.122/32,52.154.170.209/32,52.154.170.229/32,52.154.170.243/32,52.154.170.26/32,52.154.170.28/32,52.154.170.88/32,52.154.170.96/32,52.154.171.0/32,52.154.171.150/32,52.154.171.196/32,52.154.171.205/32,52.154.171.235/32,52.154.171.250/32,52.154.171.44/32,52.154.171.70/32,52.154.171.87/32,52.154.172.2/32,52.154.60.82/32,52.190.37.160/32,52.224.16.221/32,52.224.16.229/32,52.224.19.152/32,52.224.20.174/32,52.224.20.181/32,52.224.20.186/32,52.224.20.190/32,52.224.20.193/32,52.224.20.203/32,52.224.20.204/32,52.224.20.223/32,52.224.20.227/32,52.224.20.249/32,52.224.21.19/32,52.224.21.20/32,52.224.21.23/32,52.224.21.27/32,52.224.21.4/32,52.224.21.49/32,52.224.21.51/32,52.224.21.53/32,52.224.21.55/32,52.224.21.61/32,52.242.224.168/32,57.152.72.128/32,20.8.252.26/32,20.61.142.192/32,20.54.224.39/32,20.13.44.19/32,172.199.55.212/32,132.164.209.198/32,20.166.171.150/32,4.207.220.92/32,68.219.152.220/32,40.127.154.196/32,172.168.115.250/32,172.169.28.184/32,4.150.142.218/32,128.203.132.152/32,64.236.118.43/32,48.217.212.89/32,52.186.37.211/32,52.188.89.106/32,135.234.221.112/32,74.179.232.116/32,4.201.220.8/32,4.201.197.203/32,4.201.206.133/32,4.201.125.59/32,4.201.141.71/32,172.168.53.53/32,172.168.254.119/32,172.168.227.120/32,172.169.60.134/32,4.249.216.104/32,172.169.177.131/32,48.217.23.236/32,4.156.154.107/32,48.217.129.210/32,20.232.51.46/32,4.255.35.121/32,20.253.59.76/32,52.250.46.221/32,172.193.245.229/32,20.112.58.44/32,20.253.96.199/32,40.82.218.203/32,4.237.244.80/32,20.175.232.228/32,4.172.49.103/32,20.250.51.113/32,4.226.40.135/32,20.170.75.54/32,4.182.10.198/32,20.101.17.173/32,20.71.69.210/32,20.216.200.223/32,20.40.147.172/32,20.100.136.36/32,20.100.140.155/32,20.77.146.108/32,20.49.129.236/32,13.86.35.212/32,20.118.11.251/32,18.97.9.168/29,18.97.14.80/29,18.97.14.88/30,98.85.178.216/32,85.208.96.0/24,85.208.97.0/24,85.208.99.0/24,185.170.167.0/24,185.191.171.0/24

IPv6-Adressen:

2607:6bc0::/48,2001:4860:4801:10::/64,2001:4860:4801:12::/64,2001:4860:4801:13::/64,2001:4860:4801:14::/64,2001:4860:4801:15::/64,2001:4860:4801:16::/64,2001:4860:4801:17::/64,2001:4860:4801:18::/64,2001:4860:4801:19::/64,2001:4860:4801:1a::/64,2001:4860:4801:1b::/64,2001:4860:4801:1c::/64,2001:4860:4801:1d::/64,2001:4860:4801:1e::/64,2001:4860:4801:1f::/64,2001:4860:4801:20::/64,2001:4860:4801:21::/64,2001:4860:4801:22::/64,2001:4860:4801:23::/64,2001:4860:4801:24::/64,2001:4860:4801:25::/64,2001:4860:4801:26::/64,2001:4860:4801:27::/64,2001:4860:4801:28::/64,2001:4860:4801:29::/64,2001:4860:4801:2::/64,2001:4860:4801:2a::/64,2001:4860:4801:2b::/64,2001:4860:4801:2c::/64,2001:4860:4801:2d::/64,2001:4860:4801:2e::/64,2001:4860:4801:2f::/64,2001:4860:4801:30::/64,2001:4860:4801:31::/64,2001:4860:4801:32::/64,2001:4860:4801:33::/64,2001:4860:4801:34::/64,2001:4860:4801:35::/64,2001:4860:4801:36::/64,2001:4860:4801:37::/64,2001:4860:4801:38::/64,2001:4860:4801:39::/64,2001:4860:4801:3a::/64,2001:4860:4801:3b::/64,2001:4860:4801:3c::/64,2001:4860:4801:3d::/64,2001:4860:4801:3e::/64,2001:4860:4801:3f::/64,2001:4860:4801:40::/64,2001:4860:4801:41::/64,2001:4860:4801:42::/64,2001:4860:4801:44::/64,2001:4860:4801:45::/64,2001:4860:4801:46::/64,2001:4860:4801:47::/64,2001:4860:4801:48::/64,2001:4860:4801:49::/64,2001:4860:4801:4a::/64,2001:4860:4801:4b::/64,2001:4860:4801:4c::/64,2001:4860:4801:4d::/64,2001:4860:4801:4e::/64,2001:4860:4801:50::/64,2001:4860:4801:51::/64,2001:4860:4801:52::/64,2001:4860:4801:53::/64,2001:4860:4801:54::/64,2001:4860:4801:55::/64,2001:4860:4801:56::/64,2001:4860:4801:57::/64,2001:4860:4801:58::/64,2001:4860:4801:59::/64,2001:4860:4801:60::/64,2001:4860:4801:61::/64,2001:4860:4801:62::/64,2001:4860:4801:63::/64,2001:4860:4801:64::/64,2001:4860:4801:65::/64,2001:4860:4801:66::/64,2001:4860:4801:67::/64,2001:4860:4801:68::/64,2001:4860:4801:69::/64,2001:4860:4801:6a::/64,2001:4860:4801:6b::/64,2001:4860:4801:6c::/64,2001:4860:4801:6d::/64,2001:4860:4801:6e::/64,2001:4860:4801:6f::/64,2001:4860:4801:70::/64,2001:4860:4801:71::/64,2001:4860:4801:72::/64,2001:4860:4801:73::/64,2001:4860:4801:74::/64,2001:4860:4801:75::/64,2001:4860:4801:76::/64,2001:4860:4801:77::/64,2001:4860:4801:78::/64,2001:4860:4801:79::/64,2001:4860:4801:7a::/64,2001:4860:4801:7b::/64,2001:4860:4801:7c::/64,2001:4860:4801:7d::/64,2001:4860:4801:80::/64,2001:4860:4801:81::/64,2001:4860:4801:82::/64,2001:4860:4801:83::/64,2001:4860:4801:84::/64,2001:4860:4801:85::/64,2001:4860:4801:86::/64,2001:4860:4801:87::/64,2001:4860:4801:88::/64,2001:4860:4801:90::/64,2001:4860:4801:91::/64,2001:4860:4801:92::/64,2001:4860:4801:93::/64,2001:4860:4801:94::/64,2001:4860:4801:95::/64,2001:4860:4801:96::/64,2001:4860:4801:97::/64,2001:4860:4801:a0::/64,2001:4860:4801:a1::/64,2001:4860:4801:a2::/64,2001:4860:4801:a3::/64,2001:4860:4801:a4::/64,2001:4860:4801:a5::/64,2001:4860:4801:a6::/64,2001:4860:4801:a7::/64,2001:4860:4801:a8::/64,2001:4860:4801:a9::/64,2001:4860:4801:aa::/64,2001:4860:4801:ab::/64,2001:4860:4801:ac::/64,2001:4860:4801:ad::/64,2001:4860:4801:ae::/64,2001:4860:4801:b0::/64,2001:4860:4801:b1::/64,2001:4860:4801:b2::/64,2001:4860:4801:b3::/64,2001:4860:4801:b4::/64,2001:4860:4801:b5::/64,2001:4860:4801:b6::/64,2001:4860:4801:c::/64,2001:4860:4801:f::/64,2600:1f28:365:80b0::/60

Dazu empfehle ich noch eine weitere Liste von blocklist.de.

#Friendica, #Server, #Bot, #Crawler, #Scraper, #SEMrush, #nftables
www.blocklist.de -- Fail2Ban-Reporting Service (we sent Reports from Attacks on Postfix, SSH, Apache-Attacks, Spambots, irc-Bots, Reg-Bots, DDos and more) from Fail2Ban via X-ARF.

Info anonsys.net: Erweiterung des Schutzkonzeptes

Habe mir die Logs der letzten beiden Tage angesehen und dabei festgestellt, dass es doch einige Bots/Crawler gibt die keinen oder einen falschen User Agent mitschicken. Diese fallen bei meinem bisherigen Blockkonzept natürlich durch das Raster.
Aus diesem Grund habe ich jetzt in Fail2ban die Blockliste von blocklist.de eingebunden. Diese Blocklist wird alle 30 Minuten aktualisiert und ist somit stets aktuell. Das hat den Vorteil, dass mir unerwünschte Besucher, die keinen oder einen gefälschten User Agent mitschicken auch geblockt werden. Ein Blick in den Logfiles zeigt, dass diese Maßnahme zu greifen scheint. Bin gespannt, was eine längere Auswertung zeigen wird.
Somit sollten die Fedinauten hier auf anonsys.net noch besser vor Crawlern oder anderen Ungeziefern geschützt sein.

#anonsys.net #Friendica #Bots #Crawler #Fail2ban

Found out about a project with millions of randomly generated links. The author explained how #Facebook's scraping bot hit it's page 38 million times. All while the company itself claims that their bot only crawls pages that are shared on their platforms.

Why is there so much dishonesty in some hyperscaling tech companies?

Other crawlers are also listed in a short write-up by the author.

#infosec #crawler #llm #meta

Who do you think you are?

47.128.32.0 - - [18/Mar/2026:00:48:01 +0100] "GET /robots.txt HTTP/1.1" 403 239 "-" "-" 1650 4269

#Amazon #AWS Singapore.

Good on you that #CrowdSec won't immediately block on a missing user-agent, but my httpd-ACL does.

#DarkVisitors #AI #Crawler #GenAI #SocialPermissionToBurnEnergy

 I have been running iocaine on my server for a week now. During this time, 7,076,701 requests have passed through iocaine, 3,312,318 of which were identified as AI crawlers/bots. 3,741,577 requests came from crawlers/bots that got stuck in iocaine's deadly maze, consuming an infinite amount of poisoned garbage. Furthermore, 972 crawlers/bots were detected that were routed into the maze via major browsers.

All of this is managed by iocaine with just ~80 MB of memory and ~0.1% direct CPU usage. Now that’s what I call efficient! Well done, @algernon.

Let's fight back against AI crawlers and bots. Thanks to projects like iocaine, this is entirely possible, not just theory  

#iocaine #ai #llm #FckAI #FckLLMs #selfhosting #crawler #bots

Maybe AI is the response to WYSIWYG: what you see is what you claim ie WYSIWYC?
Of course, the AI industry will provide solutions, that only them can create to the flood of frauds that they have helped generating so far, by dumping models in a legal void. Who will pay for an AI premium, to protect against endless AI scams?
Just remember it all started as -for-research-

#transparency #omnibus #GDPR #EU #AI #insurance #fraud #crawler #webdev

©️ Nicolas Mouart, 2026

I have just installed iocaine 3.2.0 by @algernon and have already started successfully serving poisoned garbage to the AI agents. I love it! I especially like how simple the setup was, and how easy it was to expand my existing Caddyfile. My monthly donation is set up too. What a great project!

#iocaine #ai #llm #FckAI #FckLLMs #bot #crawler

Hallo liebe Fedinauten hier auf anonsys.net. Ab sofort wird diese Instanz vor AI- bzw. KI-Crawlern geschützt. Diese werden geblockt bzw. gebannt.

Danke @rainer für den Tipp. Habe diesen jetzt auf anonsys.net aktiviert und lasse den Filter einmal täglich aktualisieren.

Verdammt interessant ist, dass nach ca. 10 Minuten der Aktivierung des Filters bereits 128 AI-Crawler gebannt wurden:

Es werden minütlich mehr. Das ist echt Wahnsinn! 😳

Quelle: rainer.sokoll.com/?p=8353

#anonsys.net #friendica #fedinauten #ai #ki #crawler