From: Ben Goertzel (ben@goertzel.org)
Date: Tue Sep 20 2005 - 13:30:13 MDT
Richard,
I think you have made a reasonably convincing argument why AGI should
proceed via allowing a proto-AGI system to engage with a world via sensors
and actuators, and construct its own symbols to represent the world around
it. I agree that these symbols will generally not be simple, and also that
sophisticated learning mechanisms will be required to learn them.
What is not clear to me is why the "complexity" of these symbols and
learning mechanisms necessarily has to entail "complexity" in the sense of
complex systems theory (as opposed to just "complexity" in the sense of
complicatedness and sophistication). Clearly, it does in the human brain.
But you haven't demonstrated that it does in an AGI system, nor have you
really given any arguments in this direction.
Personally I suspect you are probably right and that given limited resources
complex-systems-style "complexity" probably IS necessary for effective
symbol grounding. But I don't have a strong argument in this direction,
unfortunately, just an intuition.
-- Ben
> -----Original Message-----
> From: owner-sl4@sl4.org [mailto:owner-sl4@sl4.org]On Behalf Of Richard
> Loosemore
> Sent: Tuesday, September 20, 2005 3:06 PM
> To: sl4@sl4.org
> Subject: Re: Is complex emergence necessary for intelligence under
> limited resources?
>
>
>
>
> Ben Goertzel wrote:
> > Richard,
> >
> >
> >>The answers that others offer to your questions are, pretty much: no
> >>you cannot really avoid complex systems, and mathematical verification
> >>of their friendliness is the very last thing you would be able to do.
> >>The main defining characteristic of complex systems is that such
> >>mathematical verification is out of reach.
> >
> >
> > There is no question that human mind/brain is a complex system which
> > achieves its intelligence via emergence, self-organization, strange
> > attractors, terminal attractors, and all that great stuff....
> >
> > And there is little question that emergence-based intelligence is
> > intrinsically very difficult to predict and control with a high
> > degree of reliability, thus rendering verified Friendliness an
> > unlikely outcome.
> >
> > However, these observations don't tell you much about whether it's
> > possible to use digital computers to create an intelligence that
> > DOESN'T rely critically on the emergent phenomena typically
> > associated with biological complex dynamical systems.
> >
> > Semi-similarly, the human mind/brain probably uses complex emergent
> > phenomena to add 2+2 and get 4, but, a calculator doesn't, and a
> > calculator does a better job of arithmetic anyway.
> >
> > One may argue that flexible, creative intelligence is fundamentally
> > different than arithmetic, and is not achievable within limited
> > computational resources except via complex, unpredictable emergent
> > dynamics. In fact I strongly SUSPECT this is true, but I haven't SHOWN
> > that it's true in a convincing way, and I'm not 100% convinced
> > it's true.
> >
> > If you have a strong argument why this contention is true, I'd be
> > very eager to hear it.
> >
> > On the other hand, some others seems pretty sure that the opposite is
> true,
> > and that it IS possible to achieve powerful intelligence under
> > limited resources without requiring unpredictable emergent phenomena.
> >
> > However, I haven't seen any strong arguments in this direction either.
> >
> > -- Ben Goertzel
> >
> >
>
>
>
> Ben,
>
> In reply to your question, I'll see if I can outline my argument in more
> detail than previously.
>
> [I am targetting this argument at people who actually understand what a
> complex system is: I am beyond the point of trying to educate people
> who not only do not understand, but are scornful of even making the
> effort to understand, and who repeatedly throw out false arguments based
> on caricatures of what I and the complex systems people have claimed.]
>
>
>
>
> Preliminary Remark 1: The best I am going to be able to do is offer
> convincing empirical reasons why the thesis is true; certain proof is
> beyond reach, alas. So it will always be something of a judgement call
> whether one accepts these arguments or not.
>
> *Overview.*
>
> The overall direction of the argument is going to be this: that in all
> the work on AGI to date, there has been relatively little emphasis on
> getting the "symbols" [please interpret the word loosely: these are just
> the basic representational units that encode the smallest chunks of
> knowledge about the world] to be constructed entirely without programmer
> intervention. In other words, we tend not to let our systems develop
> their symbols entirely as a result of the interaction of a learning
> mechanism with a stream of environmental input. Rather, we tend to put
> "ungrounded" symbols in, which we interpret. The argument is going to
> be that there are indications that this facet of an AGI (whatever
> apparatus allows the symbols to be fully grounded) is going to be more
> important than we suppose, and that it will introduce a great deal of
> complexity, and that that, in turn, will be impossible to avoid.
>
> *Detailed Version of this Argument*
>
> We all know that in the Good Old Days, a lot of AI folks would build
> systems in which they inserted simple tokens, labelled them "arch"
> "hand" "table" "red" and so on, then wrapped a mechanism around those
> symbols so the system could manipulate the symbols as a representation
> of a world, and as a result dispaly some intelligent behavior (like, be
> able to manipulate, and answer questions about, the placement of blocks
> on a table).
>
> What we now know is that these kinds of systems had problems, not the
> least of which was the fact that the symbols were not grounded: the
> system never created the symbols itself, so it was up to the programmer
> to interpret them manually (so to speak).
>
> What was the solution? Clearly, one part of the solution had to be to
> give the system a learning mechanism, so it could build new symbols out
> of simpler ones. Or rather, so it could build symbols from scratch, out
> of real-world raw input of some kind. Or rather... well, you can see
> that it isn't entirely clear, so let's unwrap this in a bit more detail.
>
> Question: Do we have to get the system to build all its symbols ab
> initio, from a completely blank slate? Can't we give it some starter
> symbols and ask it to develop from there? Surely it would be reasonable
> if the system had some innate knowledge of the world, rather than none
> at all? The consensus view on this questtion is that it should not be
> necessary to go all the way back to signals coming from raw nerve
> endings in eyes, ears, hands, etc, but that we should be able to put in
> some primitive symbols and get the rest of them to be generated by the
> system. (Aside: the connectionists were pretty much defined to be the
> group that broke ranks at this point and insisted that we go down to
> much deeper levels ... we will avoid talking about them for a moment,
> however).
>
> So learning mechanisms should start with some simple symbols and create
> more complex (more abstract) ones as a result of observing and
> interacting with the world. Where do we draw the line, though? Which
> primitives are acceptable, and which do we think are too high-level?
> People disagree. Many who work in the machine learning field do not
> really accept any constraints on the high-levelness of their primitive
> symbols, and are happy to get any symbols to develop into any others,
> not caring if the primitives look low-level enough that we can believe
> they escape the grounding problem.
>
> We, however (from the perspective of this essay) care about the
> grounding problem and see a learning mechanism as a way to justify our
> usage of symbols: we believe that if we find a plausible learning
> mechanism (or set of mechanisms) it would be capable of going all the
> way from very primitive sensorimotor signals, or very primitive innate
> symbols, all the way up to the most abstract symbols the system could
> ever use. If we found something as plausible as that, we would believe
> we had escaped the grounding problem.
>
> Next step. We notice that, whatever learning mechanism we put our money
> on, it is going to complicate our symbols.
>
> [Terminology Note: I will use "complicate" to mean just what it seems,
> and *nothing whatsoever* to do with "Complex" as in Complex Systems.
> Hopefully this will avoid confusion. If I use "complex" it will mean
> "as in 'complex systems'"].
>
> What do I mean by complicating our symbols? Only that if they are going
> to develop, they need more stuff inside them. They might become
> "frames" or "scripts" or they might be clusters of features, or they
> might have prototypes stored in them.... whatever the details, there
> seems to be a need to put more apparatus in a symbol if it is going to
> develop. Or, if not passive data inside the symbol (the way I have
> implied so far), then more mechanism inside it. It seems quite hard,
> for a variety of subtle reasons, to build a good learning mechanism that
> involves utterly simple, passive symbols (just a token, an activation
> level and links to other tokens) and a completely separate learning
> mechanism that is outside the symbols.
>
> Now, I might be being unfair here by implying that the reason our
> symbols became more complicated was because, and only because, we needed
> learning mechanisms. I don't think I really mean to put it that
> strongly (and it might be interesting only to future historians of AI
> anyhow). We had other reasons for making them more complicated, on of
> them being that we wanted non-serial models of cognition in which less
> power was centralized in a single learning mechanism and more power
> distributed out to the (now active, not passive) symbols.
>
> So perhaps the best way to summarize is this: we found, for a variety
> of reasons, that we were pushed toward more complicated mechanisms
> (and/or data structures) inside our symbols, in order to get them to do
> more interesting things, or in order to get over problems that they
> clearly had.
>
> This is a very subtle point, so although a lot of people reading this
> will be right along with me, accepting all of this as obvious, I know
> there are going to be some voices that dispute it. For that reason, I
> am going to dwell on it for just a moment more.
>
> Sometimes you may think that you do not need complicated, active
> symbols, and that in fact you can get away with quite simple structures,
> allied with a sophisticated learning mechanism that builds new symbols
> and connects them to other symbols in just the right, subtle way that
> allows the system as a whole to be generally intelligent. In response
> to this position, I will say that there is a trap here: you can always
> rearrange one of my complicated-symbol systems to make it look as if the
> symbols are actually simple (and maybe passive also), at the cost of
> making the learning and thinking mechanism more complicated. You know
> the kind of thing I mean: someone proposes that symbols should be
> active neuron-like things, and then some anti-neural-net contrarian
> insists that they can do the same thing with a centralised mechanism
> acting on a matrix of passive data values. We have all seen these kinds
> of disputes, so let's just cut through all the nonsense and point out
> that you can always reformulate anything to look like anything else, and
> that some types of formulation look more natural, more efficient and
> (generally) more parsimonious. So when I argue that there is a tendency
> towards more complicated symbols, I mean that the consensus intuition of
> the community is that the simplest, most parsimonious AGI systems tend
> to work with symbols that have more complicated apparatus inside them
> than simply a token plus a couple other bits.
>
> So I want to wrap up all of this and put it in a Hypothesis:
>
> *Complicated Symbol Hypothesis*
>
> "To the extent that AGI researchers acknowledge the need to capture
> sophisticted learning capabilities in their systems, they discover that
> they need their symbols to be more complicated."
>
> Corollary:
>
> "The only people who believe that symbols do not need to be
> complicated are the ones who are in denial about the need for learning
> mechanisms."
>
> ***
> Please note the status of this claim. It is not a provable contention:
> it is an observation about the way things have been going. It is
> based on much thought and intuition about the different kinds of AI
> systems and their faults and strengths. The term "complicated" is open
> to quite wide interpretation, and it does not mean that there is an
> linear growth on complicatedness as sophistication of learning mechanism
> goes up, only that we seem to need relatively complicated symbols (as
> compared with simple passive tokens with activation levels and simple
> links) in order to capture realistic amounts of learning.
> ***
>
> But now, what of it? What does it matter that we need more stuff in the
> symbols? How is this relevant to the original question about complexity
> in AGI design?
>
> To answer this, I am going to try to unpack that idea of "complicated
> symbols" to get at some of the details.
>
> When we observe humans going through the process of learning the
> knowledge that they learn, we notice that they have some extremely
> powerful mechanisms in there. I mean "powerful" in the sense of being
> clever and subtle. They seem to use analogy a lot, for example. To get
> some idea of the subtlety, just go back and look at the stuff Hofstadter
> comes up with in GEB and in Metamagical Themas. When I talk about
> "learning" I don't mean the restricted, narrow sense in which AI folks
> usually talk about learning systems, I mean the whole shebang: the
> full-up, flexible kind of learning that people engage in, where jumping
> up a level of representation and pulling analogies around seems to be
> the almost the norm, rather than the exception.
>
> Getting this kind of learning capability into an AGI is the goal, as far
> as the present discussion is concerned. Anything less is not good
> enough. I think we can all agree that there is a big gap between
> current machine capabilities and the awesome generality of the
> human system.
>
> But how to do it? How do we close that gap?
>
> At this point, everyone has a different philosophy. I look at language
> learning in a 2-to-6 year old child and I think I observe that when I
> talk to that child, I can define pretty much anything in the whole world
> by referring to loose examples and analogies, and the chold does an
> astonishing job of getting what I am saying. I even define grammatical
> subtleties that way, when teaching how to talk. But to someone like
> Chomsky, Fodor or Pinker, this entire process may be governed by a
> massive amount of innate machinery [and, yes, Fodor at least seems to
> believe that innateness does not just apply to the grammatical machinery
> but to most of the conceptual apparatus as well, with just a little
> content filling in to be done during maturation].
>
> Then there are folks who don't go for the Chomskian innateness idea, but
> who do insist that there is nothing wrong with our current ideas about
> the basic format (data and mechanisms) inside symbols, all we need to do
> is build a system that is big enough and fast enough, and connect it up
> to sufficiently much real world input (and realistic motor output
> systems) and it will develop the same rich repertoire of knowledge that
> we see in humans. These people believe that all we need to do is take
> symbols the way they are currently conceived, add some richer, improved,
> to-be-determined learning mechanisms that browse on those symbols, and
> all will eventually be well.
>
> So what do I think? I disagree with the position taken in that last
> paragraph. Now I am now going to try to focus in on exactly why
> I disagree.
>
> Imagine a hypothetical AGI researcher who first decides what the format
> of a symbol should be and then tries to hunt for a learning mechanism
> that will allow symbols of that sort to develop as a result of
> interaction with the world. Just for the sake of argument (which means
> don't ask me to defend this!) let's be really naive and throw down some
> example symbol structure: maybe each symbol is a token with activation
> level, truth value, labelled connections to some other symbols (with the
> labels coming from a fixed set of twenty possible labels) and maybe an
> instance number. Who knows, something like that.
>
> First the researcher convinces themself that the proposed system can
> work if given ungrounded, programmer-interpreted symbols. They knock up
> a system for reasoning about a little knowledge domain and show that
> given a stream of predigested, interpreted information, the system can
> come to some interesting conclusions within the domain. Maybe the
> system gets loaded on a spacecraft bound for the outer solar system and
> it can "think" about some of the ship's technical troubles and come up
> with strategies for resolving them. And it does a reasonable job, we'll
> suppose.
>
> So far, not much learning. It didn't learn about spaceraft repair from
> a textbook, or from getting out a wrenc and trying to build spacecraft,
> or from long conversations with the engineers, it was just preloaded
> with symbols and information.
>
> But the researcher is hopeful, so they start adding learning mechanisms
> in an attempt to get the system to augment itself. The idea is not just
> to get it to add to its existing knowledge, but to start with less
> knowledge and get to where it is now, by doing its own learning. We
> are, after all, on a quest to eliminate most of that preloaded knowledge
> because we want to ground the system in the real world.
>
> But as the researcher tries to devise more and more powerful kinds of
> learning mechanisms, they discover a trend. Those more powerful
> mechanisms need more complicated stuff inside the symbols. Let's
> suppose that they are trying to get analogy to happen: they find that
> the existing symbol structure is too limited and they need to add on a
> bunch of extra doohickeys that represent..... well, they don't represent
> anything that easily has a name at the symbol level, but they are needed
> anyhow to get the system to do flexible, tangled kinds of stuff that
> leads to the building of new symbols out of old.
>
> When and if the researcher tries to avoid this - tries to keep the
> symbols nice and clean, like they were before - they discover something
> rather annoying: the only way they can do this is to put more stuff in
> the learning mechanisms (outside of the symbols) instead. Keep the
> symbols clean, but make the (non-symbol-internal) learning mechanisms a
> lot more complex. And then there is even more trouble, because it turns
> out that the learning mechanism itself starts to need, not just more
> machinery, but its own knowledge content! Now there are two places
> where knowledge is being acquired: symbol system and learning engine.
> And they don't talk, these two systems. In the process of trying to
> keep the symbols clean and simple, the learning system had to invent new
> strategies for learning, and (this is the real cause of the trouble)
> some of those new learning mechanisms really seemed to be dependent on
> the content of the world knowledge.
>
> Something difficult-to-explain is happening here. It is because
> learning (knowledge acauisition and refinement) is apparently so
> flexible and reflexive and tangled in humans, that we have reason to
> believe that the (human) learning mechanism that is the generator of
> this behavior must itself involve some quite tangled mechanisms.
>
> What does this seem to imply for the design of an AGI? It seems to
> indicate that if we want a natural, parsimonious design, we are going to
> inevitably head towards a type of system in which the symbols are
> allowed to grow in a tangled way right from the outset, with knowledge
> _about_ the world and knowledge about _how to understand the world_
> being inextricably intertwined. And then, when we try to get those
> systems to actually work with real world I/O, we will discover that we
> have to add tweaks and mechanisms inside the symbols, and in the
> surrounding architecture, to keep the system stable. And sooner or
> later we discover that we have a system that seems to learn new concepts
> from [almost] scratch quite well, but we have lost our ability to
> exactly interpret what is the meaning of the apparatus inside and around
> the symbols. We might find that there is no such thing as a symbol that
> represents "cup", there is only a cluster of units and operators that
> can be used to stand for the cup concept wherever it is needed, and the
> cluster manifests in different ways depending on whether we are picking
> up a cup, describing a cup, trying to defining a cup, trying to catch a
> falling cup, trying to design an artistic looking cup, and so on.
>
> In short, we may end up discovering, in the process of trying to get a
> realistic set of human-like, powerul, recursive learning mechanisms to
> actually work, that all the original apparatus we put in those symbols
> becomes completely redundant! All the reasons the system now functions
> might actually be enshrined in the extra apparatus we had to introduce
> to (a) get it to learn powerfully and (b) get it to be stable in spite
> of the tangledness.
>
> But wait, that sounds like a presumption on my part: why jump to the
> conclusion that "all the original apparatus we put in those symbols
> becomes completely redundant"? Why on earth should we believe this
> would happen? Isn't this just a random piece of guesswork?
>
> The reason it is not a wild guess is that when the complicated learning
> mechanisms were introduced, they were so tangled and recursive (with
> data and mechanism being intertwined) that they forced the system away
> from the "simple system" regime and smack bang in the middle of the
> "complex system" regime. In other words, when you put that kind of
> reflexivity and adaptiveness in such a system, it is quite likely that
> the low level mechanisms needed to make its stable will look different
> from the high-level behavior. We *want* something that approximates our
> conventional understanding of symbols to appear in the top level
> behavior - that is our design goal - and we want enormously powerful
> adaptive mechanisms. The experience of teh complex systems community is
> that you can't start with design goals and easily get mechanisms that do
> that. And you especially cannot start with low level mechanisms that
> look like the desired high-level ones, add a soupcon of adpativeness,
> and then expect the high and low levels to still be the same.
>
> Now, the diehard AGI researcher listens to me say this and replies:
> "But I am not *trying* to emulate the messy human design: I believe
> that a completely different design can succeed, involving fairly clean
> symbols and a very limited amount of tangling. Just because humans do
> it that way, doesn't mean that a machine has to."
>
> The reply is this. As you put more powerful learning mechanisms in your
> designs, I see your symbols getting more complicated. I see an enormous
> gap, still, between the power of human learning mechanisms and the power
> of existing AGI mechanisms. There is a serious possibility that you can
> only keep a clean, non-complex AGI design by steering clear of extremely
> tangled, recursive, powerful knowledge acquisition mechanisms. And if
> you steer clear of them, you may find that you never get the AGI to
> actually work.
>
> To the complex systems person, observing the only known example of an
> [non-A]GI (the human mind), it appears to be a matter of faith that real
> learning can happen in a clean, non-complex AGI design. Their point of
> view (my point of view) is: go straight for the jugular please, and
> produce mechanisms of awesome, human level learning power, but *without*
> sending the system into Complex territory, and I will have some reason
> to believe it possible.
>
> At the moment, I see no compelling reason to believe, in the face of the
> complexity I see in the human design.
>
>
>
> I am sure I could articulate this argument better. But that is my best
> shot for the moment.
>
> Richard Loosemore.
>
>
>
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:52 MDT