Dan Fabulich wrote:
>
> Well, at that point, it seems to me that you don't have to figure out
> what, in particular, it ought to do with its simulations, since its
> primary assignment is to figure out what it ought to do. Leave questions
> like informed consent up to the expert(s).
Yes, that's what I'm hoping will happen - that the answers are an
invariant of reality and can be extracted from it. I hope. I don't know.
> I'm not familiar with your distinction of active and passive stupidity; I
> take it you mean that a rock is passively stupid, whereas the guys who
> killed Galileo were actively stupid. (A rock never makes mistakes?)
Passive stupidity is what we might call "rational" mistakes, which
includes mistakes made because of misleading evidence, inadequate
processing power, or (in the case of AIs) overly crystalline reasoning.
Active stupidity means stupidity that evolved for a specific reason;
rationalizing an incorrect political position, for example. Most of
human ethics is ringing the changes on "the ends don't justify the
means", and it exists because of our evolved tendency to make really bad
judgement calls on those questions.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/beyond.html
This archive was generated by hypermail 2b29 : Thu Jul 27 2000 - 14:04:02 MDT