You’d think these centralised LLM search providers would be caching a lot of this stuff, eg perplexity or claude.

They’re absolutely not crawling it every time they nee to access the data. That’s an incredible waste of processing power on their end as well.

In the case of code though that does change somewhat often. They’d still need to check if the code has been updated at the bare minimum.