From: Mikko Särelä (msarela@cc.hut.fi)
Date: Sun May 07 2006 - 09:24:12 MDT
Hi Eliezer,
I just watched your talk about recursion, the most important math problem
and you mentioned changing value system as one of those problems that you
have not yet been able to formulate and solve. I have some ideas, though
not rigorously in math form, which I hope might help. Or you might already
know all of these, but here goes anyways. Two different approaches, first
one tells you when you possibly should change your value/goal system and
the second one tells you how that should be aproached.
I use the term moral system as a system which describes the goals and
values, as well as the effect of values to the means of an agent. The
reason for is that a value system as such is basically uncritisizable
(Hume's Guillotine) and instead one should take both goals and means as a
whole system and allow that to be critizised and improved.
First important thing to understand is that a moral system is necessarily
composed of both values/goals and the means used to get to them. Which
means that a rational agent can look at his moral system, whenever he
learns new information and find out that for example his system is
self-defeating (which would be a bad thing).
This kind of situation requires the agent to search for a better moral
system, which can either mean changing goals, values, or means.
The next step: how should we change our goals or values, because as you
say, we get into an infinite recursion problem by doing that. The main
idea in my opinion, is that the agent should calculate the future value of
the future agents actions based on its _current_ values and goals. Thus, I
will only make such changes to my code, which result in actions and in a
mind which I, the current me, believes to maximize his own utility (not
the utility of the future mind doing the actual decisions).
One of the key things about utility theory is that utilities of two
different minds cannot be directly compared. When we are doing
modifications to our goal/value structure, we are changing the mind and
thus the utilities of the future mind and current self cannot be directly
compared. The future mind's utility can only be part of the equation, if
the current mind cares about the future mind's utility.
Does this kind of reasoning help solve the problem of infinite recursion
you were talking about? The current mind makes decisions that maximize its
utility, even though it will never as such see the results of that
maximizing. Instead it is somebody else, the future mind, who does.
--
Mikko Särelä http://thoughtsfromid.blogspot.com/
"Happiness is not a destination, but a way of travelling." Aristotle
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT