From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Tue Dec 22 1998 - 00:31:38 MST
(WAS: Failure of AI a prediction...)
Of late, the opinion has been expressed that it's okay to believe
strongly in something as long as you have evidence for it - in other
words, "dogma" describes a belief that is held strongly in the absence
of evidence. I disagree. Any strong belief is a chain. Dogma is dogma
even if the facts happen to be in correspondence with it. If an atheist
would refuse to change vis mind if presented with contrary evidence,
ve's being dogmatic.
I give credence to an opinion if it is caused by the facts, if a causal
linkage exists between that opinion and the evidence, which requires
that the removal of the evidence result in the removal of the opinion.
Any strong belief causes the opinion to persist in and of itself and
slows reaction to new information. Translation: It makes you stupid
and slow.
Consider: A parent, born a Communist, becomes a Republican on the
strength of the evidence. Vis child picks up the parent's reasoning and
repeats it verbatim. The parent is rational, the child is not - even
though they hold exactly the same beliefs. The parent would become a
Libertarian if given a copy of Reason magazine; the child would remain a Republican.
Look at CSICOP and sTARBABY! Are there psychic powers? Very likely
not. You shouldn't need to resort to distorting the evidence to prove
it. But CSICOP still managed to screw up thoroughly, solely by
believing too strongly in skepticism. They forgot that "scientific
investigation" isn't in the answer, it's how you ask the question. They
believed so strongly in a particular outcome that their experiments got
sloppy, which is bad, and then they tried to cover it up, which is
hideous. Moral: There may be strong rational evidence and excellent
moral reasons for believing, but believe too strongly and you're
screwed. Belief rots the brain.
A strong belief prevents you from factoring the negation of that belief
(at however low a probability) into your actions. I do not hold as a
point of dogma that a Singularity is possible; in fact, I factor the
(admittedly low) probability that it is *impossible* into my planning.
Strong beliefs mean being unprepared for anything you didn't believe in.
You must be able to emotionally accept both P and ~P before you can
rationally decide which is true. It is certainly acceptable to care
very strongly about the _decision_ - but it is not acceptable to care
too strongly about P.
Learn to operate without strong beliefs. Everything is a guess. I
don't care who you are; at least one thing you strongly believe is
false. Be prepared.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:50:04 MST