jestyrTwitter seeing 52% of its links removed on google after user requirement change
marswarriorI don’t think google is doing that on purpose. Search engines are supposed to remove dead links once in a while. And twitter redirecting all links to the login page, so it’s effectively making duplicate results on google, which are removed. I think that’s why google doesn’t have too many Linkedin results.
Taasz/WoofI wish they would drop dead links faster, I feel like 20% of the results I click on get 404 or the server is straight up down.
jecxjoThere is a lot of Internet to crawl
True, maybe just a report dead link button on the results page would be the way to go.
That would get abused. You click it and it gets queued to check its status and you’re right back to crawling.