Having just started playing with web #bots
It is super hostile environment for automation.
Between robots.txt, JavaScript and dynamically generated content and air #tarpits its quite apparent why the Web 4.0 (?) / folks want to pour gasoline on the web and burn it down.
"Machines not welcome" on the front door.
Ah well, back to headless browsers, and stochastic "humans" for me 😁





