TL;DR:

AI bota and AI agents desteoyed it.

Sincerely a real problem and I as a sysadmin for various www sites: I loathe them daily.

If Cisco, F5 etc could invent a way to block these bots at the firewall and load balancer level, they’d make billions.

You can do that! You just have to block known cloud service providers, known scraper ASNs, and while this is not at the firewall level, a captcha or other challenge like cloudflare or anubis.