Problems finding and installing robots.txt

[Matthew]

I’ve also posted the contents of the file here in the forum:

Blocking AI Data Scrapers in Backlight

Our next version of Backlight (v5.4.1) will include updates to our distributed robots.txt file to block AI web crawlers from indexing your site, and I’ve already updated our robots.txt documentation to include new information about it, and links to external resources. But as it’s a manual task to implement the robots.txt on your site, there’s no need to wait for the update. You can implement the new rules immediately by adding all of this to a robots.txt file at the root of your site. This lis…