I think of the Singularity as protection against Failures Of The Imagination.
Such failures can have *real* and *horrible* consequences. I have gotten
several letters from "Singularity skeptics" who think that enhanced
intelligence isn't all *that* powerful, and technology can't progress *that*
fast, and that it is therefore proper to IGNORE THE MOST ELEMENTARY
PRECAUTIONS when attempting to create a transhuman intelligence! One person
seriously proposed that you could just "pull the plug" if it started looking
dangerous! Another, on this list, said that we should just "keep it bottled
up" and only let it out "if it convinces us to". I'm sorry, but those
precautions would not suffice to keep *me* in, much less anything smarter.
Pull the plug? Hey, I'm already running on 200,000 computers around the
world. Only if it convinces us? We all know humans are impossible to fool.
If not for the "Singularity", I might not have realized that there are protein
synthesis machines which may be hooked up to the Internet... that locked-goal
Asimov circuits leak like sieves... and so on, and so on.
> > don't expect to be in charge, and it looks to me like the logic of
> > Libertarianism breaks down if you're omniscient For All Practical
> > Purposes.
>
> Sorry, I'm calling you on this one. "omniscient For All Practical
> Purposes" looks like a phrase with null semantic content to me.
> Could you please try rephrasing whatever it was you intended to
> say?
Omniscient FAPP means that you can visualize (and perfectly predict) all other
"agents" in a game-theoretic scenario. While multiple competing OFAPP agents
are questionable (halting problem?), a Power deciding whether to force-upload
humanity could well be OFAPP, game-theoretically speaking.
(Note: If external influences exist on a game scenario, OFAPP requires that
you be able to predict the external influences as well.)
-- sentience@pobox.com Eliezer S. Yudkowsky http://tezcat.com/~eliezer/singularity.html http://tezcat.com/~eliezer/algernon.html Disclaimer: Unless otherwise specified, I'm not telling you everything I think I know.