From: Matt Mahoney (matmahoney@yahoo.com)
Date: Wed Dec 02 2009 - 16:13:30 MST
Stuart Armstrong wrote:
>>The AGI knows that your fear of death is irrational, a product of evolution.
> Sigh... Any motivation, for anything whatsoever, is irrational
What is irrational or not is irrelevant to my argument. The AGI has been programmed to be truthful to humans. Therefore ve must upgrade your intelligence to ver level. Ve has a model of your mind, just like you have a model of your dog's mind that enables you to predict its actions. That model will maintain your evolved fear of death, because altering your memories, goals, or behavior would be dishonest. By killing you instantly, neither copy of you experiences any pain or fear, so your volition is satisfied, assuming ve was also programmed to satisfy your volition.
Need I point out that giving simple top level goals to Jupiter brains like honesty, or making people happy, or making paper clips can have unintended consequences?
-- Matt Mahoney, matmahoney@yahoo.com
----- Original Message ----
From: Stuart Armstrong <dragondreaming@googlemail.com>
To: sl4@sl4.org
Sent: Wed, December 2, 2009 2:32:07 PM
Subject: [sl4] Irrational motivations (was [sl4] A Thought Experiment: Thou Shalt Not Bear False Witness)
>The AGI knows that your fear of death is irrational, a product of evolution.
Sigh... Any motivation, for anything whatsoever, is irrational, in
that it does not derive from reason. Fearing death is irrational; not
fearing death is irrational; wanting an AI to make you fear/not fear
death is irrational, wanting anything at all or not wanting anything
at all is irrational. Posting on this list is irrational; refraining
from doing so is irrational; following reason is irrational. Your
whole motivation system is irrational, no matter what it is.
You might talk about certain sets of desires being inconsistent, but
you can never talk about any (non-contradictory) desire being
irrational in any absolute sense, while others are not.
Stuart
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:05 MDT