From: Mark Waser (mwaser@cox.net)
Date: Sun May 30 2004 - 13:08:35 MDT
> I tend to regard the sharp distinction between "groups" and "individuals"
> as a special case of the human way, where cognitive systems cannot
> agglomerate, and the goal systems contain numerous egocentric
> speaker-dependent variables.
I can agree that a "sharp" distinction between "groups" and "individuals" is
a special case that humans have been stuck in. However, I also believe that
you will always have (at least momentary) individuals and always have
collections of these individuals who are separate but working together (even
if they agglomerate later) and that the distinction between individuals is a
necessary one (even if it becomes far more transient and fluid in the
future).
Also, there's always the fact that you really can't have only one, monstrous
yet fully-integrated mind. The cost of maintaining consistency across a
mind scales with size/complexity. Beyond a certain size point, all you'll
be doing is maintaining consistency. You will always have non-integrated
parts (also known as individuals) - - they just will be more able to more
easily agglomerate (at the same time that more individuals will branch off).
> In particular, I worry that our being adapted
> to the individuals-groups dichotomy, and our expectation that other
> individuals will exhibit those same adaptations, can lead to incorrect
> inferences when considering AIs.
It could lead to incorrect inferences but not necessarily. Also, it is
entirely incorrect to dismiss a conclusion because the reasoning process
that arrived at it is incorrect.
> It looks to me like the challenge of
> getting a cognitive system to compute morality remains the same regardless
> of whether that physical system is called a multiplicity or a singleton -
> whether we regard it as many minds or one mind, it's the same thing.
I would agree. The reason why you want a multiplicity is because the system
is not going to be omnipotent or omniscient and it's going to have to make
decisions in the absence of perfect information - - most particularly,
decisions about it's own beliefs, value structures and goals. Now, in a
perfect world with infinite resources, the system could model the decision
being made both ways and continue to track both paths - - except then,
arguably, the system is now two systems unless the two paths inappropriately
bleed together.
My experience, from many years of system design/watching others make
mistakes, is that it is far better to have the distinction of
partitioning/individuals/distinctions built in from the start because
otherwise inappropriate bleed is inevitable. Distinctions/partitioning are
what makes it possible for us to think. Without partitions and individuals,
a bad meme could poison everything.
> Or to put it more
> pessimistically, if a course is so difficult that it can't be solved using
> an agglomerated runner, splitting up the runner into individuals can't get
> you any closer to solving the problem.
It's one hour before the destruction of the human race. I have twenty
doors - nineteen take 50 minutes to traverse and come back, one leads to the
total invulnerability of human beings in 25 minutes. Do I want one runner
or twenty?
And yes, I totally twisted your analogy from a communications problem which
I think is the wrong analogy to a goal-achievement problem.
> Humans have to imagine ways of solving problems that use many humans,
> because in the human world there is no way to use *one* *big* human.
And I argue that there is no way to *use* one big AI. There are always
costs to maintain consistency and integrity that scale with size. I AM sure
that the biggest feasible AI is MUCH bigger than a human, but there is
definitely a maximum size there and I'm pretty sure that it's smaller than
I'd be happy with.
> Suppose you have two "thermostat AIs" - that is, they have a decision
> system that employs a very simple and nonhumane way of computing
> desirability. Let's say that one AI cares only about paperclips, while
the
> other cares only about staples. If the two AIs are roughly equal, they
> might arrive at something resembling a cooperative game-theoretical
> solution and split up the solar system between them, this solution being
> preferable to the negative effects of hostilities - classic PD. The
> problem is that this doesn't protect the humans - it is better for *both*
> AIs to split any resources currently used by humanity between them.
You lost me at "one AI cares only about paperclips, while the other cares
only about staples". They both care about friendliness. If they realize
that their world views are so out of synch as to require splitting up the
solar system, their top priority should be rationalizing their world views
before they accidentally do something horribly unfriendly. You seem VERY
vested in your point of view to not see this argument as a total
non-sequitor.
> From our perspective, this might not look very different from a single AI
> with a goal system that wanted both paperclips and staples.
No, there's a huge distinction. In one case, there are two gods - one says
"I think paperclips", the other says "I think staples", and both say "uh oh,
maybe we better figure out why we don't have the same thought before we do
anything rash". In the other case, the one god buries the universe in
paperclips and staples. From my perspective, these are two very different
things.
> If you imagine that humans, in some unimaginable way, acquire a serious
> threat to hold over both superintelligences, demanding that the humans be
> treated as game-theoretical near-equals, then why wouldn't the same threat
> be holdable over a paperclips-and-staples singleton? This is what I mean
> by the apparent equivalence of groups and individuals from our
perspective.
Sure, if humans hold the ultimate power, it doesn't seem to matter whether
they hold power over a group or an individual. And your point is?
> It looks to me like it takes work to compute a humane morality, work which
> does not emerge automatically in either groups or singletons. If there's
a
> group solution I would expect there to exist a corresponding singleton
> solution.
Absolutely. No disagreement here whatsoever.
> Even if human morality is inherently group-based, the Friendly
> AI structure looks like it should work to embody our group morality in a
> single AI!
It's not morality that's group-based. It's the safest path to implementing
that morality.
> You have the problem of getting a number N of AIs to treat humans
> humanely even if humans may not be their game-theoretical equals, and this
> looks like pretty much the same problem whether N=1000 or N=1.
There are two different problems that you're conflating here. There is the
problem of getting a number N of AIs to want to treat humans humanely and
there is the problem of getting a number N of AIs to succeed in their wishes
(i.e. actually treat humans humanely). Given that we can start them off
with the goal of treating humans humanely and with the goal of maintaning
that goal, we start with the first problem already INITIALLY solved. The
problem is maintaining that solution and preventing errors. Independent
redundancy is a good solution to error prevention.
> You need an
> AI that is humane and values sentient life for its own sake.
Yes.
> If N AIs
> don't value sentient life,
we're dead.
> ** end of quote **
> I am in the middle of working out seriously complicated stuff that I am
too
> busy reworking to properly explain. Sometimes I will be able to explain
my
> reasons. Sometimes not. I am getting more and more nervous about time.
I would argue that anything that you can't easily explain, you haven't fully
worked out. I understand that there is a required initial level of
knowledge before an explanation can be understood but that knowledge should
have explanations too. I don't believe that there is anything that a single
individual can do that is so far ahead of the curve that it can't be
explained in a reasonable amount of time. Special relativity, quantum
physics, mathemetical proofs - - they all have fairly simple (if very
lengthy in the case of many recent proofs) explanations once the conceptual
leaps are initially performed. Science isn't magic. Science, by
definition, is reproducible.
I'm real nervous about time too. I'm not sure what you think you're going
for that wouldn't be sped up by a collaborative process. If you're
successfully creating an FAI all by yourself and you haven't made a mistake,
more power to you. If you're only completing your theory of Friendliness
(which doesn't have the danger of possibly advancing UFAI if shared), then
it's going to have to be explained and integrated before it will have any
effect on the world.
> Neighborly human or not, it is not a trivial task to give a specialist
> fresh advice in his own field. Figure on the attempt failing at least 95%
> of the time. Most of the time, I will have long since thought through
> everything that occurred to you, in advance, whether that is readily
> apparent or not. By all means keep trying, but if I say "I already
thought
> of that," I did, whether I have time to explain or not.
I'm not offering you fresh advice here. I understand that you think that
you've thought through this. You could probably successfully blow me off by
using those magical words "agree to disagree" but normally I wait to use
those words until we both can point to one pretty low-level fact that is the
linch-pin of both our arguments on which we disagree. I don't think that
the community is anywhere near that point on one FAI versus many.
Mark
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT