From: Matt Mahoney (matmahoney@yahoo.com)
Date: Mon Apr 28 2008 - 20:36:03 MDT
--- Stefan Pernar <stefan.pernar@gmail.com> wrote:
> Assumption:
> Intelligence is defined as an agent's ability to maximize a given
> utility function.
> Friendliness can be expressed as an agent's utility function.
>
> Conclusion:
> An agent, who's utility function is to be friendly will be friendlier
> the more intelligent it becomes.
It will be "friendlier" according to the fixed definition of
friendliness that we program into it. Friendliness is complex, so we
will surely get it wrong. However, the agent is not allowed to change
its utility function. The utility function does not get smarter along
with the rest of the agent.
This is a real threat. As the agent gains intelligence it will gain
new capabilities. Our 10 million lines of friendliness code never
failed when the AI was used to filter spam. Now it is smart enough to
control our nuclear launch system...
-- Matt Mahoney, matmahoney@yahoo.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT