Mastodawn

AI Has Created a Battle over Web Crawling
https://spectrum.ieee.org/web-crawling
#ycombinator #data_sets #fair_use #foundation_models #generative_ai #lawsuit #websites

With Robots.txt, Websites Halt AI Companies' Web Crawlers

With more websites restricting crawler bots, AI companies may soon be hurting for training data

IEEE Spectrum