The Robots Exclusion Protocol

From: Ken Meyering (ken@define.com)
Date: Thu Aug 27 1998 - 05:08:08 MDT


Derek:

I noticed that the robots exclusion file is misnamed on
www.extropy.org.

The file should be named "robots.txt" rather than "robot.txt".

I noticed that

   http://www.extropy.org/robot.txt exists and that
   http://www.extropy.org/robots.txt does not exist.

Please see:
 
http://info.webcrawler.com/mak/projects/robots/exclusion.html

My personal hope is that this archive be the exclusive archive, and
that the archives at lucifer.com and alienlogic.com be deleted.

Of course, I'm also a free speech advocate and do "own my words", so
what comes back to haunt me is my own responsibility. However, in
the interest of encouraging open discussion, it would be preferable
if there was a mechanism to say "please don't index this message".
My signature is an attempt to send this signal to robots, although
I'm not sure that this will work.

Thanks.

ken@define.com
_______________________________________________________
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">



This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:49:31 MST