Google pushes for an official web crawler standardIt's open-sourcing the tool it uses to scan robots.txt files.By J. Fingas, 07.01.2019