- A combination of standards and conventions governing web crawler behavior and search engine indexing. The REP includes the robots exclusion standard (robots.txt), crawling and indexing directives for HTML
meta
elements, the Sitemaps protocol, and therel=nofollow
microformat. ↑ rfc-editor.org/rfc/rfc9309
- Previous term: Robot
- Next term: Robots exclusion standard
- Random term: DML