What I learned from analysing thousands of robots.txt files

About this talk

The humble robots.txt file (Robots Exclusion Protocol) has been in existence for 25 years and can be hugely influential for SEO. This talk covers an analysis of its current usage by some of the top-ranking sites across key verticals, uncovering some interesting statistics and some of the most common implementation mistakes. It is increasingly important that marketing professionals ensure they are getting theirs right, as Google strives to make robots.txt an internet standard.