From: Nick Bostrom (bostrom@ndirect.co.uk)
Date: Tue Feb 23 1999 - 15:17:19 MST
Billy Brown wrote:
> I see two fatal criticisms of the idea suggested in the FAQ:
>
> First, it is mind control. Remember, posthumans are by definition fully
> sentient beings. Programming them to abide by a preordained moral code is
> no different than doing the same thing to our own children, or to each
> other. I can see no possible way to justify such an action on moral
> grounds.
I think you are confusing to issues here. (1) doing it to our
children; (2) doing it to each other. These cases are diffrent.
(1) There is nothing morally wrong in doing it to our children.
Responsible parents do it all the time. To impart good values,
behaviours and morals to our children is not just a right but a
responsibility. Without it, our children couldn't function in our
society. If we imagine, for the sake of the argument, that they could
function without such parental shaping, and also that they would be
as powerful as a posthuman will be, then we could expect horrible
things to happen; it would be like putting nukes in the hand of a
super-clever, but peevish and egomaniac toddler.
(2) Doing it to each other, on the other hand, is only allowed
through certain means such as persuation. The reason why there is a
moral difference between the two cases is that in the first case
there is no pre-existing person with violable rights. There are just
different potential persons, and we make a choice about which one
shall come into being (by choosing who we copulate with and how we
raise our children, traditionally; or by choosing the values we
program into a superintelligence, in the hypothetically). We can
assume that we only choose between sets of values that the person
would be happy with once it is in existence, and that would lead to
a life worth living. In the second case, there are pre-existing
persons, and if they don't want to be transformed by your
mind-changing methods, then you have no right to transform them. (I'm
skipping over some subtleties here that aren't relevant to my main
point.)
> Second, in the long run it won't work. You can program non-sentient
> machines to do your bidding, but that isn't what the FAQ is talking about.
> It suggests that we attempt to enforce an eternal program of mind control on
> fully sentient posthumans - for those of you who have forgotten what that
> means, I suggest you read the definition of 'Posthuman' in the same
> document. Does anyone really think such a prohibition would work on humans
> for all of eternity? Then why do we think we can do it to an entire society
> of superintelligent, self-modifying entities?
Two points: First, a being who has a certain fundamental value
*doesn't want to change it*, per definition. So it's not as if these
guys will think they are being mind-fucked and try to figure out a
way to get around it. No more than you are trying to abolish your own
survival instinct just because you know that it is an artifact of our
evolutionary past.
Second, later generations of posthumans may well be born or created
by earlier generations. They might well have the ability and the will
to propagate their fundamental values to their creations/offspring.
(If you love humans, you wouldn't choose to have a child that will
hate humans and exterminate them!) And they could presumably invent a
highly reliable mechanism for transmitting their values to the next
generation.
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:07 MST