#SEO Tip: for large-scale sites, using a noindex meta tag isn’t a good way to control crawl budget.
It can however indirectly free up crawl budget in the long-run. This is done by Googlebot focusing on other URLs on a site, which can have the long-run impact.
The best practices for improving crawl efficiency (according to Google) include:
- Consolidating duplicate content
- Block crawling of URLs using robots.txt
- Return a 404 or 410 for permanently removed pages
- Keep your sitemaps up-to-date
- Make your pages efficient to load
As a reminder, crawl budget is an area which should only be of concern to large sites. Google categorises this as sites that either have:
1. 1M+ unique pages with content that changes moderately often (once a week)
2. 10K+ unique pages with rapidly changing content (daily)
3. Sites with a large portion of their total URLs classified in GSC as ‘Discovered - currently not indexed’
This is important information to keep in mind next time you’re asked about how to influence crawl budget, and whether it should be of concern.
#searchengineoptimization #seotips #google #crawlbudget