Entity
robots.txt
A text file that instructs web crawlers which pages or files to access or ignore on a website.
术语属性
- Purpose：Control web crawler access
- Format：Text file
Entity
A text file that instructs web crawlers which pages or files to access or ignore on a website.