From: Jeff Medina (analyticphilosophy@gmail.com)
Date: Tue Feb 21 2006 - 22:16:55 MST
William Pearson wrote:
> So the maximum number of bits of Phi and hence number of known or
> provably friendly programs is bounded by the length of your starting
> program.
As Eliezer mentioned, the number is infinite, and hence unbounded
(i.e., the cardinality of the set of known or provably friendly
programs reachable from an arbitrary starting program can be
aleph-null). What you bound by choosing a particular starting program
(via its length, but also obviously varying across different
structures of the same length) is the trajectory space. The
cardinality of that space, if no probabilistic hypothesis generation
methods are used, is 1 (i.e., N copies of the same deterministic
friendliness program should make the same decisions as one another).
Eliezer wrote:
> There is an obvious infinite class of Friendly programs which are all
> the same except for containing N bits of unreachable code.
Noted. Tangent: Any such state should be unstable, unless those bits
are also somehow unknowable/undetectable, because a Friendly program
capable of recognizing it contained N bits of unreachable code should
self-modify to remove those N bits, on pain of wasting resources that
could be used to do something reachably-Friendly. And direct deletion
wouldn't work, because that implies the code is reachable for
modification; we get around that by simply copying all reachable code
to create a new, equally Friendly, non-bit-wasting sibling of the
original.
Eliezer wrote:
> 2) If the 'Friendly' invariant includes dependencies on outside
> physical processes which can affect the approval of new programs, e.g.,
> coherent extrapolated volition contains a dependency on the cognitive
> states of humanity, then the bound does not formally hold because new
> bits are absorbed from the environment.
The trajectory of a program through state space will depend on its
inputs/environment, like any context sensitive automaton would (and
should). The bound will still hold as long as one takes program input
into account, and I can't think of any reason one wouldn't do so. Hard
to be Friendly if you can't gather any data on the universe & its
inhabitants toward whom your Friendliness is supposed to refer.
-- Jeff Medina http://www.painfullyclear.com/ Community Director Singularity Institute for Artificial Intelligence http://www.intelligence.org/ Relationships & Community Fellow Institute for Ethics & Emerging Technologies http://www.ieet.org/ School of Philosophy, Birkbeck, University of London http://www.bbk.ac.uk/phil/
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:55 MDT