From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Sep 18 1999 - 19:30:14 MDT
den Otter wrote:
>
> > From: Eliezer S. Yudkowsky <sentience@pobox.com>
> > Okay, suppose I accept this. I still don't see why any SI will
> > automatically drop off all its emotions *except* selfishness and then
> > you think this is a good thing. That's really where you lost me.
>
> Well, unless suicide is the big meaning of life (and the SI actually
> gives a hoot about such things), it will need to retain
> self-preservation in its mental structure. You need to be alive to do
> things. I'm not saying that the SI will drop "all of its emotions", btw.
> More likely it would modify them, and/or get rid of some, and/or add new
> ones.Or perhaps it *would* get rid of them, but that's just one of many
> options.
Okay, but it wouldn't need to have an all-consuming "survival" goal,
much less a goal of "growth". For example, the only reason why it would
have a survival goal could be to fulfil its ultimate supergoal of "Help
den Otter have ultimate fun." Or even "Survive long enough to upload
den Otter, then commit suicide."
Are these goals impossible or unstable? Why? If one set of goals is
intrinsically irrational, it's not a very long step to conclude that a
single goal is maximally rational and all SIs converge to it, and it's a
pretty short step from there to Externalism.
And if they're not irrational - *I* think they are, but then I'm an
Externalist; you aren't - why not a Servant AI?
> > But why not? There's one emotional whim, "must find the truth". And
> > another emotional whim, "stay alive". And another, "be happy". In me
> > the first one is stronger than the other two. In you the two are
> > stronger than the one. I don't get the part where your way is more
> > rational than mine, especially if all this is observer-dependent.
>
> What it all comes down to is scoring "satisfaction points", correct?
No. Precisely and exactly wrong. How many satisfaction points does the
concept of "scoring satisfaction points" gain you?
> That's what drives us on. Even you. You set more or less random
> goals, and reaching those goals will give you satisfaction (or remove
> frustration) and thus generate points.
Maybe that's the way the human mind is built - to treat projected future
satisfaction as present satisfaction, and present satisfaction as the
arbiter of choices - but surely you won't deny me the right to alter
that? To unbind the semantics of the mind? There are cognitive
elements in this game other than simple satisfaction, and they can
affect each other, build self-sustaining loops that arbitrate choices
directly. To you, I suppose, it would seem an unnatural and illogical
knot, something like circular logic, a twisted takeover by elements of
rationality that should rightfully be serving emotion. But I did it.
I'm free.
> Ok, now that we've determined
> that we're in the same race, we can compare strategies. Some
> strategies will get you more points than others. For you, finding the
> truth (or creating a SI which may do that) is obviously worth a lot of
> points, but, and this is a key issue, it isn't the *only* thing that
> is worth points (to you).
It's the only thing whose point-worth I choose to acknowledge - whose
future point-worth is worth points to me. This kind of arbitrary power
over the present may be harder to enforce, as you note below - but it
isn't too hard with respect to the mere projection of emotional satisfaction.
> Other, more "mundane" activities, like
> watching _Buffy the Vampire Slayer_, are also a source of points.
I know that all work and no play will cause me to chip and crack like a
cheap plate. Me especially, emphasis on "Special"... Perhaps I've
struck and rationalized the wrong happy medium, influenced by the
built-in point-allocation systems, but that's hardly the same as
accusing me of engaging in non-Singularitarian activities.
Besides, even _Buffy_ serves my purposes directly. I've been
experimenting with daimones, _Aristoi_ style. Also known as LPs
(Limited Personalities), morphs, and switchable self-symbols.
Apparently watching this show for nine hours straight (I was on
vacation) has interesting side effects.
> If all goals are essentially equal (in that they are ways to get
> satisfaction/reduce discomfort), then the logical thing is to
> pick a strategy that will earn you maximal emotional currency.
Big *if*. If all goals are essentially equal, you can program an SI
with whatever goals you like and get a Servant. If all goals are
essentially equal, than earning minimal emotional currency is precisely
as logical as earning maximal emotional currency.
You've made a big deal out of asking me to acknowledge that my
Externalist principles are driven simply by an emotional desire to find
the truth, some "higher meaning" to life. Why can't you acknowledge
that maximizing projected future emotional satisfaction is driven by a
desire to maximize future satisfaction? If you deny me the
representational binding between my mental model of the satisfaction of
unspecified External goals and whatever specific External goals may or
may not exist, I can deny you the binding between your future
satisfactions and your current desires to maximize projected future satisfactions.
> Short but dangerous thrills like smoking Crack cocaine or
> spawning SIs may get you a lot of points at once, but then
> you die and that's it, no more points. If you settle (temporarily)
> for lesser pleasures (less points), but survive (much) longer,
> you will ultimately gain a lot more points, perhaps even
> an infinite amount (the happy god scenario). You win!
Once again, you're assuming a semantic binding between future and
present that I've quite deliberately broken. Suppose that I take over
your brain, and through dark neurosurgeries cause you to be tremendously
satisfied, emotionally satiated, by hapless servitude to me. Would you
value this more than the simple pleasure of, oh, eating ice cream? Even
though the absolute emotional points are orders higher? No, because you
view it as "invalid" satisfaction points. The binding is broken. I've
simply done the same thing on a much larger scale.
> To recap, the best thing for you would be to drop your
> relatively dangerous goal of creating an ASI, and compensate
> for the loss of (anticipation-fun) points by concentrating on
> other fun stuff, and/or modifying your mental structure (if
> possible) so that "finding the truth" is linked to uploading
> and living forever. This new memeplex could fill the void left
> by the previous one ("Singularity at all costs").
But this talk of voids and filling uninterests me. My goal isn't
emotional satisfaction, it's doing whatever is rationally correct.
> I don't have any "grand" goals myself, really. Ok, so of course
> I want to become immortal, god-like, explore all of reality and
> beyond etc. etc., but that's hardly the meaning of life. I don't
> really care about the meaning of life. Maybe it exists, maybe
> not. Maybe I'll find it, maybe not. Who cares; I've done just fine
> without it so far, and I don't see why this would have to change
> in the future.
You still believe in yourself and your world. You are not sufficiently confused.
> > I particularly don't get the part where an SI converges to your point of
> > view. And if it doesn't converge, why not build an SI whose innate
> > desires are to make you happy, and everyone else who helped out on the
> > project happy, and everyone on the planet happy, thus hopscotching the
> > whole uploading technological impossibility and guaranteeing yourself a
> > much larger support base?
>
> Is this a joke? I mean, *this* coming from *you*?
It would be a joke, coming from me. The question I keep asking is - why
isn't it coming from you? I mean, why *don't* you believe this?
> See above. Ultimately goals are about avoiding "bad" feelings and
> generating good ones. Punishment & reward. Reason is just a tool
> to get more reward than punishment. We should always keep this
> in mind when selecting "random" goals.
I wish I was a clever guy like you and could know stuff like that. I'd
feel so much more confident if I thought I understood the Universe that
well. [Sarcasm.]
> I'm "pretty sure", but of course I can never be "completely sure"
> either. It doesn't matter; I can't really lose with my approach.
Of course you can. You can dither over uploading and sabotage AI and
get eaten by grey goo as a result, when all along SIs would have
converged to friendliness and freely upgraded you to godhood. Your
philosophy implies actions, and actions can *always* go wrong. I
acknowledge the risks I'm taking; why can't you?
> I'd
> have forever to decide what to do next. You on the other hand
> are betting everything on a force that, once released, is totally
> beyond your control.
Ask me if I think there are *any* controllable forces in the game...
You put too much faith in the essential benevolence of the Universe.
> My way (careful, gradual uploading) is
> better because it would allow me to have some measure of
> control over the situation. Having control increases the
> chances of achieving your goals, whatever they may be.
So does acknowledging reality. I might also go for gradual uploading,
if I thought it was practically possible. It ain't.
> > I'll certainly agree with that part. Like I said, the part I don't get
> > is where you object to an AI.
>
> The AI may decide that it doesn't want me around. The AI is
> only useful from *my* pov if it helps me to transcend, and I
> don't think that's very likely; I'm probably just a worthless bug
> from *its* pov.
Once again - if you, den Otter, program the AI with a servile point of
you, why do *you*, all-goals-are-equal guy, think it would converge to
anything else?
And I note you still haven't answered my future-past emotional-unbinding
demonstration - you know, the "This scenario is worth 20 points" business.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/tmol-faq/meaningoflife.html
Running on BeOS Typing in Dvorak Programming with Patterns
Voting for Libertarians Heading for Singularity There Is A Better Way
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:05:12 MST