From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Jun 08 2002 - 15:16:28 MDT
Eugen Leitl wrote:
>
> I wish you'd stop pointing people to an unreadable >800 kBytes document,
> and would answer a few direct questions on list. As you notice, I'm not
> criticing any details of your design, because I don't know them. I'm just
> describing the constraints any controlling arbiter (your system included)
> must follow.
You asked your questions, I gave the answers, you kept on writing exactly
the same things. If you don't believe my answers that's one thing, but
don't accuse me of refusing to answer your questions.
At least that's how it looks from over here.
>>The question of how a seed AI team can avoid exerting undue influence
>>over humanity's future - and when you're talking about humanity's
>>entire future, ANY personal influence is "undue influence" - is one of
>>the deep questions to which Friendly AI is intended as an answer. It
>
> Creating a seed AI and succeeding is clearly an influence over humanity's
> future, and a rather large and irreversible one at that. Talking about
> "deep questions" and "undue influence" in this context takes some nerve.
But it doesn't have to be *our* seed AI - that is, it doesn't have to be a
seed AI where anyone would or could look at it and say, "Hm, this seed AI
was designed by a programmer who enjoyed reading Terry Pratchett novels." I
confess to not liking scenarios where Earthgoing intelligent life is
destroyed - but to steer around such catastrophes does not, to me, seem to
consist of exerting undue influence on the future. If you accuse me of
intending to set my personal stamp on the future, making it in any sense
"my" future rather than humanity's through abuse of the role I intend to
play, then that is a very serious accusation and I should like to hear it
backed up.
> Sorry, as a human you're too limited to allow decisions on this scale. I
> don't trust anybody's judgement on this.
Nor do I, of course. That is why a Friendly AI must be more than a proxy
for the programmers.
> The good part about enforcing consensus is that none of the players is
> omnipotent. I'd rather not see a manmade god make a moral error of any
> magnitude, thankyouverymuch.
A group, an individual, what difference? A moral system is a moral system
regardless of how many nodes it's implemented on. Either you can figure out
the stuff that moral systems are made of, or you can't. Arguing about
whether it should be implemented on N nodes or 1 node is anthropomorphic.
I wouldn't like to see a manmade god make a moral error of any magnitude
either. What's your point?
>>you assume an AI is a proxy, despotic or otherwise, there will be no
>>moral answers to how you can pass on morality to an AI. A Friendly AI
>
> Morality is not absolute --> there is no Single Golden Way to Do It.
Ah, I see. Is that an objective principle or is it just your personal
opinion? If you were to tell a Friendly AI: "I think there is no Single
Golden Way to Do It", would you be stamping your personal mark on the future
or stating a truth that extends beyond the boundaries of your own moral
system? What is the force that makes your statement a message that is
communicable to me and the other members of the Extropians list? By what
criteria should I decide to accept this statement, and by what criteria
should the other members of the Extropians list decide that they wish me to
accept this statement? If a Friendly AI watched you make the statement,
watched me decide whether to accept the statement, and watched the other
list members decide whether they wanted me to accept or reject your
statement, and the Friendly AI thereby absorbed a set of moral principles
that enabled it to see the force of your statement as an argument, would you
then feel a little more inclined to trust a Friendly AI?
>>needs to embody those moral principles that govern what kind of
>>morality it is legitimate to pass on to an AI, not just the moral
>>principles its creators happened to have at the AI's moment of
>>creation. This inherently requires delving into questions of
>>cognition about morality and not just moral questions of the sort
>>usually argued over. The question of how to build an AI that is an
>>independent moral philosopher is *not* equivalent to, and in some
>>cases barely touches upon, the question of what sort of moral material
>>a Friendly AI should initially be given as an experiential learning
>>set.
>
> Meaningless metalevel description, not even wrong. Tell me where you
> derive your action constraints from to feed into the enforcer. <--- that's
> a genuine, very answerable question.
Not until you define what an "action constraint" is; as for "enforcer" it
sounds distinctly un-Friendly-AI-ish. You seem to have a very limited view
of what a Friendly AI is, consisting of some kind of mathematical constraint
set or Asimov Laws. Start with an analogy to a human moral philosopher
instead of a steam shovel. It will still be a wrong analogy but at least
you won't be using the Spock Stereotype to guide your vision of how AIs
work. An Extropian futurist should know better.
>>I've heard you sketching your scenarios for how a team of five uploads
>>who deliberately refuse intelligence enhancement will forcibly upload
>>millions or billions of other humans and scatter them over the solar
>>system before the Singularity begins. I will simply leave aside the
>
> 1) we don't know whether a hard edged Singularity will occur naturally,
> so please don't try to precipitate one just because we can.
A hard edged Singularity will occur naturally. You have agreed with me on
this on several occasions. Smarter intelligence leads to still smarter
intelligence; positive feedback loops tend to focus on their centers. Are
you now retracting your agreement?
> 2) that was an ad hoc scenario. Idiotic enough to not even put it up to
> generic discussion.
Then WHAT IS YOUR SCENARIO? Kindly present an alternative!
> 3) please kindly remove the "forcibly upload" from it, okay? Since you're
> the one who's mentioning "misrepresenting" so often.
Very well. I assume you stand by your previous statement that those who do
not upload voluntarily will be dogfood. However, this is a perfectly
defensible moral position, albeit a strongly negative scenario.
> Let's agree to disagree about what is hard and what is easy. If there is a
> Singularity I think making its early stage kinetics less fulminant
HOW?
> is considerably easier than building a (f|F)riendly singleton, and doesn't
> run risks of blighting this place to boot.
Give me a scenario. Incidentally, a Friendly seed AI is not necessarily a
singleton. All we know is that this would appear to be one of the options.
A seed AI team does not build a Friendly singleton. It builds a seed AI
that must, if necessary, be able to serve as a singleton. This propagates
back ethical considerations such as "no undue influence". It is not a
decision on the part of the seed AI team to pursue a singleton-based future;
that kind of decision should not be made by human-level intelligences.
> If you don't want your viewpoint misrepresented, I suggest you drop vague
> accusations towards me and address where I've supposedly wronged you
> in a direct discussion with concrete information.
Very well; the gross misrepresentation is this statement:
>>> as soon as a very small group codifies
>>> whatever they think is consensus at the time into a runaway AI seed, and
>>> thus asserts its enforcement via a despot proxy.
This is a gross contradiction of our declared intentions and a gross
misrepresentation of the systemic process of building a Friendly AI.
> Critical seed AI research in machina is potentially extremely dangerous,
> and needs to be regulated for the duration of the vulnerability window.
> While far from being an optimal solution, I'm not yet aware of a better
> one.
How long a vulnerability window? What is it that closes the vulnerability
window? How, specifically, does it happen? How does uploading, which
requires an enormous amount of computing power and neuroscience knowledge,
occur *before* the successful construction of a seed AI?
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:14:41 MST