From: Stathis Papaioannou (stathisp@gmail.com)
Date: Tue Apr 08 2008 - 07:11:24 MDT
On 08/04/2008, Nick Tarleton <nickptar@gmail.com> wrote:
> Something like Pascal's Mugging (
> http://www.overcomingbias.com/2007/10/pascals-mugging.html ) is a
> problem even without someone trying to mug you. It may be that there
> is some way (say, a magic button hidden in the middle of the Boötes
> Void, hence the title) to allow 3^^^^3 (or any other very large but
> low-algorithmic-complexity number) times as many happy posthumans to
> exist than could ever exist in the universe without 'magic' - or, more
> generally, for most utility functions there may exist 'magic' to
> create much much more utility than would otherwise be possible. This
> is extremely unlikely, but it seems doubtful that it would be unlikely
> enough to have lower expected utility than the default course. This
> suggests that devoting massive resources to searching for magic is a
> convergent subgoal for any utility maximizer whose utility function
> admits of Really Big Numbers, and that even a Friendly AI, if it fit
> that description, would do so, possibly even to the exclusion of
> supporting existing humans, or at least diverting a majority of the
> resources that could go to existing humans.
This is along similar lines to the discussions in OB, but what reason
is there to ignore the possibility that searching for 3^^^^3 units of
magical utility will lead to at least 3^^^^3 units of magical
disutility?
-- Stathis Papaioannou
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT