Learn what the robots.txt file does and how to audit it for technical SEO problems.

You will learn the basic directives:

  • User-agent
  • Disallow
  • Allow
  • Sitemap

Example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php/

You will also learn the important difference between blocking crawling and preventing indexing.

PreviousNext