David Chisnall (*Now with 50% more sarcasm!*)Web design in the early 2000s: Every 100ms of latency on page load costs visitors.
Web design in the late 2020s: Let's add a 10-second delay while Cloudflare checks that you are capable of ticking a checkbox in front of every page load.
Jernej Simončič �@david_chisnall I don't even care about Cloudflare (and Anubis) checks – those at least rarely last more than a few seconds. What I loathe are the throbbing placeholders that seem to be everywhere now, causing simple text pages to load slower than similarly-looking pages (once the content renders) loaded on dial-up.
Eggs now in different baskets.@jernej__s @david_chisnall Don't get me started on Anubis.
I was browsing with SeaMonkey and wanted to find out if it was possible to customise/edit a search engine in SeaMonkey. So I followed a link to the mozillazine forums.
Using SeaMonkey I could NOT get past the Anubis check, it just hung and never completed.
Maybe these systems could also check the browser strings and be clever enough to realise that a SeaMonkey user might have a genuine reason to visit the mozillazine website?
@the_wub They check user-agent and challenge anything that claims to be Mozilla (because that's what the majority of bots masquerade as).
Also, weird that Seamonkey can't pass it – I just tried with Servo, and it had no problems.
Magnus Ahltorp@jernej__s @the_wub Every graphical web browser claims to be Mozilla.
@jernej__s 1/n SeaMonkey is still based on an ancient version of the Firefox codebase.
I love the email client, and the browser has the tabs in the right place, but the browser fails to work on features in modern web sites. Ones that do not fall back gracefully.
I presume that this is what causes Abubis challenges to fail when using SeaMonkey.
I can get into Mozillazine without any Anubis challenge appearing using Netsurf. Which has a limited implementation of javascript.
@jernej__s 2/n So I installed NoScript in SeaMonkey to see if it is a javascript issue.
With javascript turned off I get this message.
"Sadly, you must enable JavaScript to get past this challenge. This is required because AI companies have changed the social contract around how website hosting works. A no-JS solution is a work-in-progress."
So being blocked now from using an add-on to protect myself from malicious scripts on websites.
OK so I will now whitelist Mozillazine.
@jernej__s 3/n
Aha! A message I did not get the last time I got stuck trying to get into Mozillazine using SeaMonkey.
"Your browser is configured to disable cookies. Anubis requires cookies for the legitimate interest of making sure you are a valid client. Please enable cookies for this domain."
(But SM is set to accept all cookies.)
So in order for websites to protect themselves from AI scraping users have to reduce the level of security they are prepared to accept as safe when browsing.
@jernej__s n/n
Or to go through processes of whitelisting all of the relevant sites that you wish to visit as safe so that Anubis can validate your browser whilst otherwise disabling cookies and javascript for other sites.
Or just go find other sites to visit that do not assume you are a bot and block you from viewing content.
As regards SM being set to accept cookies and Anubis not recognising that maybe my pihole is blocking something that Anubis expects to find in a valid client?
@the_wub Sadly, when you get 5000 requests per second from residential IPs (each IP doing 10-20 requests, all using user agents from legitimate browsers), there's very little other things you can do. This is not an exaggeration, that's what was happening at a client that has a web server hosting about 50 sites for their projects – they were getting hit with that several times per week, bringing the whole server down until we implemented Anubis.
@jernej__s I understand the battle.
The protective measures taken though should not make things more dangerous for users.
Unless, of course the internet is nearing the end of the path as a free and open source of information.
In which case, what does it matter.
@jernej__s OK. Here is the rub.
I get challenged twice trying to get into the Mozillazine forums.
1) now it lets me pass but I have to make sure the link from my search engine is the https on NOT the http link.
2) I get to the Mozilla landing page with a list of links to the forums.
These links are all http.
I cannot get past the Anubis challenge unless I alter the link to the https version.
Now finally logged into the forums with SeaMonkey.
@jernej__s As the OP began this thread saying how we have to wait for tens of seconds for Cloudflare like challenges and I have spent a lot more than this amount of time this morning sorting one problem with Anubis for one site for one browser.