From: Anders Sandberg (asa@nada.kth.se)
Date: Tue Feb 12 2002 - 03:47:45 MST
On Mon, Feb 11, 2002 at 01:13:59PM -0800, hal@finney.org wrote:
>
> "Archipelago is the only answer to ruling a population of trillions,
> who own a million different cultures, mores and histories."
> He shrugged. "It is simple: an artificial intelligence - a mechal
> brain, if you will - exists and mediates things. It knows each and
> every citizen personally and orchestrates their meetings with others,
> communications and so on in order to avoid irreconcilable conflict.
> Beyond that, it stays out of sight, for it has no values, no desires
> of its own. It is as if every person had their own guardian spirit,
> and these spirits never warred, but acted in concert to improve
> people's lives."
...
> This is reminiscent of Eliezer's "Friendly AI" scenario, where the AI
> acts benevolently to keep things running smoothly. Which is better,
> this "tyranny of condescension", or a world in which people are free,
> even though they bring harm to others?
It is not obvious that the "tyranny" is even possible as stated. It is
the old problem of information again; while the AI may (somehow) get
around the usual problem that the ruling entity does not have full
information about the desires and actions of its subjects, it still is
forced to deal with an immense optimization problem that has to be
solved in real-time. It is not clear whether this would involve
preventing any bad encounter (and thus ending up with the risk of ending
up in a local optimum leading to a far worse long-term outcome) or
allowing some to happen in order to prevent worse problems in the
future. To some extent this is what people do all the time - just think
of a good host or hostess at a party - but it is not obvious at all that
this can be scaled up arbitrarily.
It is also not explained whether it is possible to ignore the advice of
the AI in the above scenario. If it is, I see no ethical problem with it
at all. If it overrides the free behavior of the citizens, then there
are clearly problems with it (unless one is an untilitarian, in which
case the issue becomes whether this system actually manages to maximize
utility).
-- ----------------------------------------------------------------------- Anders Sandberg Towards Ascension! asa@nada.kth.se http://www.nada.kth.se/~asa/ GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:12:19 MST