I am trying to improve the blocking of search engines from indexing any part of my sites. I do not want to password protect my sites as this would be a barrier to some of my friends/family. I have upgraded to Backlight 6, but I am still using older templates. I have not noted any problems with these older templates in Backlight 6 or previous versions of Backlight.
I have just removed the robot.txt files from my two sites (collinsnet.us and thewilderness.us). I have added the meta command to the Home, About, Info, and Contact page templates in the Page Copy: Meta description fields on each of these templates.
Have I done this correctly? Will this accomplish my objective?
I have read Matt’s posts on using robot.txt but am not sure how to “block” my entire site and not just (for example) the thumbnails folders. Would this be the proper syntax (assuming the Meta approach would not work, and I need to use robot.txt)?
User-agent: *
Disallow: /backlight/collinsnet.us
followed by
User-agent: Google-Extended
Disallow: /
and other search engine specific User-agents?
Thanks for helping!