From synaptic plasticity to motor control

The talk discusses some fundamental nonlinear control tools relevant to the conference theme, including adaptive nonlinear control, dynamic system identification, contraction theory, and distributed convergence.

This talk will provide an overview of techniques developed in my group to learn a model of the feasible space of solutions using machine learning technique. This is the key to generate robust and flexible robotic controllers capable of adapting their strategies in the face of unexpected changes in the environment. I will present solutions to allow instantaneous reactions to perturbation, mimicking humans’ immediate response in the presence of danger. This finds applications for compliant control during human-robot collaborative tasks and for performing sports with moving targets.

There has been rapid progress in the application of reinforcement learning to difficult problems such as playing video games from raw-pixels, controlling high-dimensional motor systems, and winning at the games of Go and Poker. These recent advances in reinforcement learning have been built on top of the backpropagation of error algorithm. Large networks are key to success, and to train these networks effectively reinforcement algorithms typically backpropogate either TD-errors (e.g. DQN) or policy gradients (e.g. TRPO and A3C) or both (e.g. DDPG). Whether the brain employs deep learning algorithms remains contentious, and just how the brain might implement approximations of the backprop algorithm remains a mystery. I will review recent progress in deep reinforcement learning and argue that these results further compel us to figure out whether the brain implements some form of deep learning. I will then discuss recent work aimed at understanding how backprop might be approximated by cortical circuits.

Populations of neurons display an extraordinary diversity in the types of problems they solve and behaviors they display. Techniques have recently emerged that allow us to create networks of model neurons that solve tasks of similar complexity. Examples include the FORCE method, a novel technique that harnesses chaos to perform computations. We demonstrate the direct applicability of FORCE training to spiking neurons by training networks to mimic various dynamical systems in addition to reproducing more elaborate tasks such as input classification, storing sequences, reproducing the singing behavior of songbirds, and recalling a scene from a movie. Post-training network analysis reveals behaviors that are consistent with electrophysiological data, such as the stereotypical decrease in voltage variance upon input presentation, reproducing firing rate distributions from songbird data, and reproducing locations of incorrect recall in sequence replay. Finally, we demonstrate that theta oscillations are critical for both learning and recall of episodic memories.

Providing the neurobiological basis of information processing in higher animals, spiking neural networks must be able to learn a variety of complicated computations, including the generation of appropriate, possibly delayed reactions to inputs and the self-sustained generation of complex activity patterns, e.g. for locomotion. Many such computations require previous building of intrinsic world models. The talk will show that and how spiking neural networks may solve these different tasks. Firstly, I will present constraints under which classes of spiking neural networks lend themselves to substrates of powerful general purpose computing. The networks contain dendritic or synaptic nonlinearities and have a constrained connectivity. They can then be combined with learning rules for outputs or recurrent connections. I will show that this allows to learn even difficult benchmark tasks such as the self-sustained generation of desired low-dimensional chaotic dynamics or memory-dependent computations. Furthermore, I will show how spiking networks can build models of external world systems and that the acquired knowledge can be used to control the systems. References: D. Thalmeier, M. Uhlmann, B. Kappen, and R.-M. Memmesheimer, Universal computation with spikes, PLoS Comput. Biol. 12:e1004895 (2016). L.F. Abbott, B. DePasquale, R.-M. Memmesheimer, Building functional networks of spiking model neurons. Nat. Neurosci. 19:350-355 (2016).

To plan and control movement, the brain needs to construct a model of the non-linear dynamics of the body in response to neuro-muscular commands. How a network of spiking neurons can learn such a model, by adjusting interconnection weights in a biologically plausible way, is still unresolved. As an advance in this direction, we propose a local and stable learning scheme, Feedback-based Online Local Learning Of Weights (FOLLOW) [Gilra and Gerstner, arXiv:1702.06463]. We show that the learning scheme is uniformly stable with the error going to zero asymptotically, under reasonable approximations. We apply the FOLLOW scheme to enable a network of interconnected spiking neurons to learn the dynamics of a linear, non-linear or chaotic example system. We also make the network learn the dynamics of a simplified two-link arm, then use the network to control the arm, to draw a desired shape on a wall.

Error-backpropation is the classical algorithm that has been boosting deep learning and that, since its publication (Rumelhart, Hinton & Williams, Nature 1986), has inspired the search for its implementation in the brain. One of the main obstacles withstanding its direct implementation was that the error calculation needs separate neurons that should modulate the synaptic strength on other neurons. We show that this problem is overcome when neurons are considered with multiple dendrites. We postulate that in the apical dendrite of pyramidal neurons a neuron-specific error is calculated that tells how much the top-down input deviates from the bottom-up input in that neuron. What the local Martinotti feedback cannot subtract from the top-down input remains as prediction error in the apical tree. This prediction error modulates the neuronal activity of the pyramidal neuron and drives plasticity in the synapses delivering the bottom-up input to its basal tree. We also highlight a new theory according to which the neuronal dynamics follows equipotential lines in a high-dimensional manifold, and plasticity acts to lower the potential energy until the output error is minimized. The theory can be mapped to the cortical microcircuits that calculate the prediction errors.

The human brain is a dynamical system whose extremely complex sensor-driven neural processes give rise to conceptual, logical cognition. Understanding the interplay between nonlinear neural dynamics and concept-level cognition remains a major scientific challenge. Here I propose a mechanism of neurodynamical organization, called conceptors, which unites nonlinear dynamics with basic principles of conceptual abstraction and logic. It becomes possible to learn, store, abstract, focus, morph, generalize, de-noise and recognize a large number of dynamical patterns within a single neural system; novel patterns can be added without interfering with previously acquired ones; neural noise is automatically filtered. Conceptors may help to model how conceptual-level information processing emerges naturally and robustly in neural systems.

I will discuss two recent results from our Lab:
1. A generic model for excitatory and inhibitory neurons on layer 2/3 of a cortical column learns under STDP to extract and represent repeatedly occurring pattern components from their input stream. A key feature of the model is data-based divisive inhibition for pyramidal cells, rather than strict lateral inhibition as in simpler winner-take-all circuit models. It turns out that the precise form of inhibition has substantial influence on the emergent computational function of the network. Details can be found in a paper that just appeared: Z. Jonke, R. Legenstein, S. Habenschuss, and W. Maass. Feedback inhibition shapes emergent computational properties of cortical microcircuit motifs. Journal of Neuroscience, 37(35):8511-8523, 2017.
2. A generic recurrent network of excitatory and inhibitory spiking neurons learns under STDP to associate previously learned memories that are each encoded by assemblies of neurons. A remarkable aspect is that this simple model reproduces new data from neural recordings in the human brain (Ison et al., 2015), (De Falco et al., 2016). These data suggest that the neural code for combined memories differs in the human brain from what had been assumed in previous neural network models and theory.A preprint of our paper (Pokorny et al., 2017) on this will be posted on my homepage http://www.igi.tugraz.at/maass/publications.html before the beginning of this workshop.