Re: poly: The singleton hypothesis

Nick B. writes:
>> >The slave owners lacked certain essential technologies. For example,
>> >they could not read the slaves' minds, and they could not simply
>> >program the slaves to totally embrace whatever values the slave
>> >owners wanted them to have. ... peoples' brains will no longer be
>> >blackboxes where all sorts of unexpected things can happen; they
>> >will be computational structures amendable to direct
>> >manipulation by anybody in power at time zero.
>>
>> There is a vast difference between a single world government and a
>> such a totalitarian power. Even when it in principle possible to read
>> and modify brains, it will be at first very expensive.
>
>Do you agree that this technology could be a strongly stabilizing
>factor once it's cheap? And how could it be very expensive if it
>requires nanotechnology? If you have one brain-scan machine, you can
>use nanotech to cheeply give you (almost) any number you want.

It's just wrong to think that nanotech makes everything too cheap
too worry about costs. *Relative* costs are the ones that will matter.
Relative to the costs of doing other useful things, what are the costs
of ensuring that all agents values and actions don't have consequences
which might threaten the totalitarian power? Is there an simple cheap
query which one can make of an agent which can easily allow one to tell
about such threats? What if agent values are encoded in very distributed
ways, and described in terms which are not very directly related to whether
or
not to threaten the totalitarian power? What if their planned acts
threaten the power even if their values don't? How much do you have to
analyse the consequences of their actions that the agents didn't forsee?
And how often do you have to physically rip apart agents to ensure that
they are faithfully responding to external queries about internal states?

You might be able to design agents from the ground up in such a way that
it was cheap to monitor their loyalty to the totalitarian power. But
the cost of destroying all other agents would seem an enormous loss.
Received on Sat May 2 03:27:23 1998