Why is #twitter not properly identifying itself as a bot when trying to scrape my website? (69.12.56.0/21 is AS63179 is Twitter)
Could it be cause they're a malicious party training an #aibot?
(This is extremely low-intensity, but based on the combination of this specific UA and the pages they're trying to reach, I've seen them before, coming in from residential proxies.)
The funny thing is that bots identifying as bots and observing robots.txt would actually be allowed to reach those particular pages.







