From: Robin Hanson (hanson@econ.berkeley.edu)
Date: Mon Jul 20 1998 - 13:22:37 MDT
Dan C. wrote:
>> >Since the SI will be vastly more intelligent than humans, IMO we may not
>> >be able to comprehend its motivations, much less predict them. The SI will
>> >be so smart that its actions are constrained only by the laws of physics,
>> >and it will choose a course of action based on its motivations.
>>
>> Why do you assume such a strong association between intelligence and
>> motivations? It seems to me that intelligence doesn't change one's
>> primary purposes much at all, though it may change one's tactics as one
>> better learns the connection between actions and consequences.
>
>Human motivation is less complex than the motivations of ants?
You lost me here.
>Robin, the reason I produced the list of motivations and actions was
>to attempt to provide specific examples. Can you reccomend a way for
>me, or another human or group of humans or construct of humans (short of
>an SI) to reliably assign probabilities to that list?
Dan had written:
>...
>M: SI wants to maximize its power long-term
>A: SI sends replicator probes to in all directions.
>
>M: SI wants to die.
>A: SI terminates itself. ...
It seems to me that the motivations of future entities can be predicted
as a combination of
1) Selection effects. What motivations would tend to be selected for
in a given situation?
2) Legacy motivations. Descendants of current creatures will likely
retain much of their motivations, translated to a new context.
Wanting to die isn't favored by selection or legacy, except for certain
translation possibilities. Spatial colonization will be selected for,
and has lots of legacy pushing for it as well.
Increases in intelligence allow creatures to anticipate selection effects,
accelerating their effects of creatures who prefer to be selected.
Increases in intelligence also raise the possibilities of creatures
attempting to integrate their otherwise disparate legacy preferences
under simpler unifying principles. Otherwise, I don't see how increases
in intelligence should lead us to expect much change in motivations.
Robin Hanson
hanson@econ.berkeley.edu http://hanson.berkeley.edu/
RWJF Health Policy Scholar, Sch. of Public Health 510-643-1884
140 Warren Hall, UC Berkeley, CA 94720-7360 FAX: 510-643-2627
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:49:22 MST