den Otter wrote:
You have to be kidding. Short of grabbing random derelicts off the
street, there's simply no way you could do that. Or were you planning
to grab random derelicts off the street?
>
> ----------
> > From: Eliezer S. Yudkowsky <sentience@pobox.com>
>
> > Has it occurred to you that if the first uploads *are* charitably
> > inclined, then it's *much* safer to be in the second wave? The first
> > uploads are likely to be more in the nature of suicide volunteers,
>
> The first uploads would no doubt be animals (of increasing complexity),
> followed by tests with humans (preferably people who don't grasp
> the full potential of being uploaded, for obvious reasons).
> Of course,
> the strictest possible safety measures should be observed at all
> times. Only when these tests are concluded with satisfactory results,
> should the (actual) synchronized upload procedure be executed.
If only the Straumers had stamped "Thank you for observing all safety
precautions" on that ancient archive! This seal of Solomon would have
kept the Blight from getting out.
You're dreaming.
> > especially when you consider that a rough, destructive, but adequate
> > scan is likely to come before a perfect, nondestructive scan.
>
> Well, IMHO the scan version of uploading is utterly useless from the
> individual's point of view (copy paradox and all that), and I certainly
> wouldn't waste any time on this method. In fact, I'm just as opposed
> to it as I am to conscious AI.
So. No backups.
> > You're staking an awful lot on the selfishness of superintelligences.
>
> I'm simply being realistic; when you realize how incredibly slow,
> predictable and messy humans will be compared to even an early
> SI, it is hard to imagine that it will bother helping us. Do we "respect"
> ants? Hardly. Add to that the fact that the SI either won't have our
> emotional (evolutionary) baggage to start with, or at least can modify
> it at will, and it becomes harder still to believe that it would be
> willing to keep humans around, let alone actively uplifting them.
This holds just as true for uploaded humans, except that being all "messy" we're even less likely to wind up in a state remotely resembling what we started out with.
> Why would it want to do that? There is no *rational* reason to
> allow or create competition, and the SI would supposedly be the
> very pinnacle of rationality. It's absurd to think that a true SI would
> still run on the programming of tribal monkey-men, which are
> weak and imperfect and therefore forced to cooperate. That's why
> evolution has come up with things like altruism, bonding, honor
> and all the rest. Nice for monkey-men, but utterly useless
> for a supreme, near-omnipotent and fully self-contained SI. If
> it has a shred of reason in its bloated head, it will shed those
> vestigal handicaps asap, if it ever had them in the first place.
> And of course, we'd be next as we'd just be annoying microbes
> which can spawn competition. Competition means loss of control
> over resources, and a potential threat. Not good. *Control* is good.
> Total control is even better. The SI wouldn't rest before it had
> brought "everything" under its control, or die trying. Logical, don't
> you think? Forget PC and try to think like a SI (a superbly powerful
> and exquisitely rational machine).
*You* try to think like an SI. Why do you keep on insisting on the preservation of the wholly human emotion of selfishness? I don't understand how you can be so rational about everything except that! Yes, we'll lose bonding, honor, love, all the romantic stuff. But not altruism. Altruism is the default state of intelligence. Selfishness takes *work*, it's a far more complex emotion. I'm not talking about feel-good altruism or working for the benefit of other subjectivities. I'm talking about the idea of doing what's right, making the correct choice, acting on truth. Acting so as to increase personal power doesn't make rational sense except in terms of a greater goal.
And, speaking of IA motivations, look at *me*. Are my motivations human? Not by your definition. That's all from some trivial little twist in the quantitative levels of abilities... nothing qualitative.
> Or are you hoping for an insane Superpower? Not something you'd
> want to be around, I reckon.
No, *you're* the one who wants an insane SI. Selfishness is insane. Anything is insane unless there's a rational reason for it, and there is no rational reason I have ever heard of for an asymmetrical world-model. Using asymmetric reflective reasoning, what you would call "subjectivity", violates Occam's Razor, the Principle of Mediocrity, non-anthropocentrism, and... I don't think I *need* any "and" after that.
> Synchronized uploading would create several SIs at once, and though
> there's a chance that they'd decide to fight eachother for supremacy,
> it's more likely that they'd settle for some kind of compromize.
Or that they'd merge.
> > Maybe you don't have the faintest speck of charity in your soul, but if
> > uploading and upgrading inevitably wipes out enough of your personality
> > that anyone would stop being cooperative - well, does it really make
> > that much of a difference who this new intelligence "started out" as?
> > It's not you. I know that you might identify with a selfish SI, but my
> > point is that if SIs are *inevitably* selfish, if *anyone* would
> > converge to selfishness, that probably involves enough of a personality
> > change in other departments that even you wouldn't call it you.
>
> To transcend is to change, dramatically, about that I have no doubt.
> So what, I'm not who I was when I was, say, 2 or 5 or 12. In some
> aspects I'm the polar opposite of what I was then, but I still consider
> myself to be me. Drugs can change the mind temporarily almost
> beyond recognition, and while you dream your dream persona can
> be quite different from the "real" you, both outside and inside, yet
> you still feel that it's "you". So, I'm not too worried about ascension-
> related personality changes, as long as I remain conscious and
> reasonably in control while it happens. Sooner or later, the monkey-
> man will have to pass on.
> > > -Stopping you from writing an AI wouldn't be all that hard, if I really
> > > wanted to. ;-)
> >
> > Sure. One bullet, no more Specialist. Except that that just means it
> > takes a few more years. You can't stop it forever.
>
> Maybe not forever, but perhaps long enough to tip the balance in favor
> of uploading.
See below.
> > All you can do is
> > speed up the development of nanotech...relatively speaking. We both
> > know you can't steer a car by selectively shooting out the tires.
>
> No, but you *can* slow it down that way.
Of course you can. But does it really matter all that much, to either of us, whether a given outcome happens in ten years or twenty? What matters is the relative probabilities of the outcomes, and trying to slow things down may increase the probability of *your* outcome relative to *my* outcome, but it also increases the probability of planetary destruction relative to *either* outcome... increases it by a lot more. You can't selectively shoot the Yudkowsky who wants to bring about a pure Singularity without also killing the Yudkowsky who's working on neurosurgical intelligence enhancement and the Yudkowsky who's studying AI motivations.
> > > You can run and/or hide hide from nanotech, even
I see. You're going to simultaneously upload six million people?
> > > fight it successfully, but you can't do that with a superhuman
> > > AI, i.e. nanotech leaves some room for error, while AI doesn't (or
> > > much less in any case). As I've said before, intelligence is the
> > > ultimate weapon, infinitely more dangerous than stupid nanites.
> >
> > Quite. And an inescapable one. See, what *you* want is unrealistic
> > because you want yourself to be the first one to upload,
>
> That's *among* the first to upload, which is something else entirely.
> Well, yes of course I want that; after all, the alternative is to meekly
> wait and hope that whoever/whatever turns SI first will have mercy on
> your soul. If I had that kind of attitude I'd be a devout Christian, not
> a transhumanist. Wanting to be among the first to upload is morally
> right, if nothing else, just like signing up for suspension is morally
> right, regardless whether it will work or not. It's man's duty (so to
> speak) to reject oppression of any kind, which means spitting death
> in the face, among other things. AI could very well be death/
> oppression in sheep's clothing (which reminds me of the movie
> "Screamers", btw, with the "cute" killer kid), so we should treat it
> accordingly.
>
> > which excludes
> > you from cooperation with more than a small group
>
> Theoretically a group of almost any size could do this, more or
> less SETI-style (but obviously with a good security system in
> place to prevent someone from ascending on the sly). I'm not
> excluding anyone, people exclude *themselves* by either not
> caring or giving in to defeatism, wishfull thinking etc.
> > and limits your
> > ability to rely on things like open-source projects and charitable
> > foundations.
>
> Why would it limit that ability? Even if you'd want to keep your project
> secret you could cooperate with people and organizations which
> might somehow advance your cause, without them ever knowing.
> Happens all the time. Anyway, it *isn't* secret. On the contrary,
> it's all over the web.
I think you overestimate the tendency of other people to be morons. "Pitch in to help us develop an open-source nanotechnology package, and we'll conquer the world, reduce you to serfdom, evolve into gods, and crush you like bugs!" Even "Help us bring about the end of the world in such a way that it means something" has more memetic potential than that. There's a *lot* more evolution devoted to avoiding suckerhood than lemminghood.
> What *they* want is unrealistic because they want to
> > freeze progress.
>
> Who is "they"?
The ones who want to keep Life As We Know It around indefinitely.
> > Both of you are imposing all kinds of extra constraints. You're always
> > going to be at a competitive disadvantage relative to a pure
> > Singularitarian
>
> What's a "pure" Singularitarian anyway, someone who wants a
> Singularity asap at almost any cost? Someone who wants a
> Singularity for its own sake?
Yep.
> > or the classic "reckless researcher", who doesn't demand
> > that the AI be loaded down with coercions, or that nanotechnology not be
> > unleashed until it can be used for space travel, or that nobody uploads
> > until everyone can do it simultaneously, or that nobody has access to
> > the project except eight people, and so on ad nauseam. The open-source
> > free-willed AI project is going to be twenty million miles ahead while
> > you're still dotting your "i"s and crossing your "t"s.
>
> Just because something is easier, doesn't mean that it's the right
> thing to do. Instead of trying to find an intelligent solution, you're
> actively contributing to the problem; it's like punching holes in an
> already sinking ship (and actually taking great pride in it too), while
> instead you should be looking for, or building, a life raft.
Why, thank you! I rather like that metaphor. The one about punching holes in the ship, I mean, not the part about the life raft. There is no life raft. Humanity *will* sink. That is simply not something subject to alteration. Everything must either grow, or die. If we embrace the change, we stand the best chance of growing. If not, we die. So let's punch those holes and hope we can breathe water.
> > A-priori chance that you, personally, can be in the first 6 people to
> > upload: 1e-9.
> > Extremely optimistic chance: 1%
>
> Why 6? It could be 600 or 6000 for all I care, as long as uploading
> happens simultaneously. If, say, half of all serious transhumanists
> decided to go for it [uploading], we'd each stand more than a
> 1% chance, simply because 99.99...% of the world's population
> lacks vision.
Believe me, den Otter, if I were to start dividing the world's population into two camps by level of intelligence, we wouldn't be on the same side. You may be high-percentile but you're still human, not human-plus-affector.
And, once again, you are being unrealistic about the way technologies develop. High-fidelity (much less identity-fidelity) uploading simply isn't possible without a transhuman observer to help. I could be wrong about this, but I really don't think I am - not from looking at the technology. Again, those 6000 are suicide volunteers. Any uploadee is a suicide volunteer until there's an SI (whether IA or AI) to help. There just isn't any realistic way of becoming the One Power because a high-fidelity transition from human to Power requires a Power to help.
> > Extremely pessimistic chance that AIs are benevolent: 10%
> >
> > Therefore it's 10 times better to concentrate on AI.
>
> Well, see above. Besides, a 90% chance of the AI killing us
> isn't exactly an appealing situation. Would you get into a
> machine that kills you 90% of the time, and gives total,
> unprecedented bliss 10% of the time? The rational thing is
> to look for something with better odds...
Yes, but you haven't offered me better odds. You've asked me to accept a 1% probability of success instead. I think you're just favoring that 1% probability because it *looks* like something you can increase, while the probability of a friendly SI is absolute. Well, but IMHO the probability of a friendly IA is absolute. And even if you can increase the probability, you aren't going to be able to increase it to 10%. I think you're taking a very sentimental and unrealistic approach to navigating the future.
> Humanity (those not in the project) would benefit too from
> the mass upload approach, because either [uploaded] people
> retain the key parts of their original personality, which means
> that the "good guys" (fanatical altruists) would protect the
> mehums, while the "bad guys" probably wouldn't risk a lethal
> conflict over the issue and some compromize would be made,
> OR personalities would change beyond recognition in which
> case humanity wouldn't be any worse off than in the case of
> an AI transcension. Conclusion: survival chances likely better
> than 10% for *everyone*, and up to 50% for those directly
> involved, depending on speed of AI development, nanotech
> etc. A long shot, but the only one that makes sense.
Well, this is where I fundamentally disagree with you. It comes down to that "OR" statement you made. I think that the second branch is much more likely - and humanity wouldn't be any worse off than in the case of an AI Transcend, but said Transcend becomes much less probable relative to nuclear war or grey goo problems. What you're overlooking is that what counts isn't just the relative desirability of the *results* of the two scenarios, but the relative *probability* of the two scenarios compared to known undesirable scenarios. An IA Transcend involves so many constraints, preconditions, and necessities that it would need a *huge* desirability advantage to be preferable to an AI Transcend.
As far as I can tell, your evaluation of the desirability advantage is based solely on your absolute conviction that rationality is equivalent to selfishness. I've got three questions for you on that one. First: Why is selfishness, an emotion implemented in the limbic system, any less arbitrary than honor? Second: I know how to program altruism into an AI; how would you program selfishness? Third: What the hell makes you think you know what rationality really is, mortal?
And I think an IA Transcend has a definite probability of being less desirable than an AI Transcend. Even from a human perspective. In fact, I'll go all the way and say that from a completely selfish viewpoint, not only would I rather trust an AI than an upload, I'd rather trust an AI than *me*. And I mean that from a strictly selfish standpoint! Human minds are too goddamn messy and they are NOT designed to tolerate architectural changes. The risk of destructive insanity is far, far higher. No matter what it is you want to preserve in an SI, it has a better chance of being there - zero, in my opinion, but a more plausible zero - if you put it in a clean, elegant AI instead of a messy human conviction. I think you're expressing a faith in the human mind that borders on the absurd just because you happen to be human.
> What needs to be done: start a project with as many people as
> possible to firgure out ways to a) enhance human intelligence
> with available technology, using anything and everything that's
> reasonably safe and effective
*Laugh*. And he says this, of course, to the author of "Algernon's Law: A practical guide to intelligence enhancement using modern technology." Which is the *other* problem with steering a car by shooting out the tires... Taking potshots at me would do a lot more to cripple IA than AI. And, correspondingly, going on the available evidence, IAers will tend to devote their lives to AI.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/tmol-faq/meaningoflife.html
Running on BeOS Typing in Dvorak Programming with Patterns
Voting for Libertarians Heading for Singularity There Is A Better Way