From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Mar 20 2002 - 17:46:19 MST
"Smigrodzki, Rafal" wrote:
>
> Good. I thought so too. But the question remains - will the FAI
> develop goals objectively following from the panhuman layer but not
> intellectually accessible to unenhanced humans? Would you foresee having
> to acquiesce to the FAI's moral guidance as an act of faith, in hope of
> eventually reaching its exalted plane (after gaining the equivalent of a
> couple hundred IQ points)?
I don't see what's wrong with this. One possibility is that you might just
gulp down a couple hundred (thousand) IQ points directly, thus obviating
"eventually". Another possibility is that after chatting with the FAI for a
couple of hours, you would have heard several major revelations about
morality - i.e., incremental advances small enough to be accepted relative
to your current knowledge - and would therefore be willing to credit that
the FAI knew something about morality. Based on the inductive
generalization from observed abilities of the FAI, and deductive reasoning
about the probable capabilities of a transhuman, you might then be willing
to assign a high probability that moral guidance received from the FAI is
accurate. I guess you could call that "faith" but I don't see why it
couldn't be governed by the Bayesian Probability Theorem just like
everything else.
> Let's say 80% of humans not only want to be simply left alone to do
> their God-fearing stuff but also insist on striking down the infidels. 15%
> want to upload and go to the stars, and they wouldn't lift a finger to
> prevent malignant nanos (not really dangerous to their highly evolved
> selves) from eating the unenlightened brutes down below. 5% want to be
> nice to everybody, don't want to kill the high-techs, but would prohibit
> self-replicating nano development if needed to assure the continued
> ability of low-techs to, you guessed it, do their stupid God-fearing
> stuff.
>
> I know you cannot foresee the results of an SAI's analysis of such a
> situation but I would think it might be difficult for any single human to
> understand and accept its suggestions as objective truth (except if by a
> lucky coincidence he/she happens to have superhuman idea(l)s already).
The 80% of humans who wish to "strike down the infidels" all wish to strike
down different infidels, and would, if asked for a moral justification of
their various hatreds, ground their justifications in moral reasonings that
rest on different objective falsehoods.
Earth, the Galactic Luddite Preserve, may ban nanotechnology (i.e., the
summed volition of the Pedestrians may be a ban); I see no reason to ban it
anywhere else.
Perhaps humans will tend to accept what an FSI says to the extent that they
want the FSI to talk to them, as opposed to wanting to be left alone.
> How do you verify the excellence of an SAI's ideas, and
> differentiate them from a high-level FoF?
We verify the excellence using the same philosophy we used to arrive at all
the moral content we gave the FAI to start with. If an FAI develops the
ability to do moral thinking that is so advanced as to be incomprehensible
to us mere humans, it would presumably be accompanied by the ability to toss
out a few trivial tidbits that are tiny enough incremental advancements over
current understanding for us mere moral bozos to grasp them and be
enormously impressed.
> #### I tend to think that qualia might be the unavoidable
> accompaniment of information processing. This is in a way equivalent to
> saying that consciousness is a feature of every information processor,
> albeit to a varying degree. My PC has the amount of consciousness (if you
> excuse such a vague expression) perhaps equivalent to a beetle. An SAI
> would have much more of this precious quality. Whether there would be an
> analogy to human emotions would depend on the amount of similarity between
> the panhuman template and the SAI's motivational mechanisms, but I do think
> that qualia of some sort would be present.
Okay. I disagree.
> Eliezer:
>
> Within a given human, altruism is an adaptation, not a subgoal. This is
> in the strict sense used in CFAI, i.e. Tooby and Cosmides's "Individual
> organisms are best thought of as adaptation-executers rather
> than as fitness-maximizers."
>
> Rafal:
>
> What is an adaptation if not an implementation of a subgoal
> of a goal-directed process?
Human subgoals are far more context-sensitive than evolution's subgoals.
Evolution is limited to induction. If it didn't happen to your ancestors at
least once, then from evolution's perspective it doesn't exist. A subgoal
can be adjusted in realtime; adaptions are fixed except over evolutionary
time.
> Eliezer;
>
> I think this is a common misconception from the "Age
> of Game Theory" in
> EP. (By the "Age of Game Theory" I mean the age
> when a game-theoretical
> explanation was thought to be the final step of an
> analysis; we still use
> game theory today, of course.) Only a modern-day
> human, armed with
> declarative knowledge about Axelrod and Hamilton's
> results for the
> iterated Prisoner's Dilemna, would employ altruism
> as a strict subgoal.
>
> Rafal:
>
> I disagree. Even ancient philosophers used imaginary
> scenarios (like the ring of Gyges) to derive a condemnation of selfishness
> as incompatible with orderly social life. Game theory gave us a more
> rigorous foundation for such reasoning but it is not absolutely necessary
> for reaching the conclusion.
The philosophers rationalized an impulse which they already possessed. That
humans possess altruism as an indepedent adaptation tells us that there were
at least some individually beneficial aspects of altruistic behavior which
were not produced by independent a priori reasoning in at least some humans
at least some of the time. Otherwise there would have been no selection
pressure. That some philosophers may have been able to rationalize
evolution's conclusion after the fact does not mean that they rationalized
all of the conclusion, that they would have reasoned to that conclusion
independently, or that Greek philosophers were common in the ancestral
environment.
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:13:03 MST