Setting a noindex flag in Backlight 6

Backlight 4, 5 & 6 Support
#1

I am trying to improve the blocking of search engines from indexing any part of my sites. I do not want to password protect my sites as this would be a barrier to some of my friends/family. I have upgraded to Backlight 6, but I am still using older templates. I have not noted any problems with these older templates in Backlight 6 or previous versions of Backlight.

I have just removed the robot.txt files from my two sites (collinsnet.us and thewilderness.us). I have added the meta command to the Home, About, Info, and Contact page templates in the Page Copy: Meta description fields on each of these templates.

Have I done this correctly? Will this accomplish my objective?

I have read Matt’s posts on using robot.txt but am not sure how to “block” my entire site and not just (for example) the thumbnails folders. Would this be the proper syntax (assuming the Meta approach would not work, and I need to use robot.txt)?

User-agent: *
Disallow: /backlight/collinsnet.us

followed by
User-agent: Google-Extended
Disallow: /

and other search engine specific User-agents?

Thanks for helping!

#2

Backlight ships with a prepared robots.txt file, and I recommend using that as your baseline.

We also have documented robots.txt pretty thoroughly:

And I have a thread here on the forum where I’ve been updating our default robots.txt file in real time:

If you want to block everything, then I think this:

User-agent: *
Disallow: /