β
#seotip What is Robots .txt?
It is the manual for robots aka user agents on how to crawl the pages of your website. Your site can only have one /robots.txt
They are designed in groups of user agents you are targeting:
π User-agent: * means all
Directories of what they can access or not access
With 3 main instructions:
- Allow
- Disallow
- Sitemap
π Submit sitemaps helps the crawler to find any sitemaps that are associated with your website
Important to write the full URL for your sitemap .xml