Lagrangian multipliers and neural learning?

From: Christian Szegedy (szegedy@or.uni-bonn.de)
Date: Mon May 14 2001 - 11:55:37 MDT

I am not an expert of neural networks and AI, but I am
interested in them know some of the most basic ideas there.
I am curious about the opinion and thoughts of the gurus
in this list on the (possible) connection between neural
learning and Lagrangian optimization.

Recently, worked on some very new and cute
optimization algorithms using sophisticated Lagrangian
relaxation techniques, which seems to be a topic
becoming more and more recognized in areas outside
the classical nonlinear optimization to which they belongs
historically. These methods based on a duality between
two types of variables: the original ones and the
so called "Lagrangian multipliers" which may be familiar
to some of you from elementary analysis courses.
Generally they are much more useful than a simple
optimization method learnt there. There is huge theory
about them with deep results and a lot of applications.

An interesting thing about them (to me) is that they seem
to be closely related to neural learning algorithms, since
this lagrangian multipliers can be interpreted as a
weighting of some components of the dual of the objective
function. So during the algorithms, this dual variables
punish the "bad" variables by increase their weight, so
the algorithm can learn that they are "dangerous" and
should not increase (or decrease) them.

Of course this description was highly surficial and far
from being scientific.

I also know that AI consists of much more than neural
learning, but this is also considered to be a possible
component of a future AI.

My question: is this connection recognized and exploited
by the researchs of AI?
Do they use the results and strength of the theory and
methods of Lagrangian relaxation explicitly or implicitely?

(It is well possible that this connections are already
discovered and heavily used by AI experts: I am a newbie
in this terrain.)

Thanks, Christian Szegedy

This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 08:07:39 MST