Adaptive Behavior, 1 (3)

Avoiding looming objects (possible predators) is essential for animals' survival. This article presents a neural network model to account for the detection of and response to a looming stimulus. The generation of an appropriate response includes five tasks: detection of a looming stimulus, localization of the stimulus position, computation of the direction of the stimulus movement, determination of escape direction, and selection of a proper motor action. The detection of a looming stimulus is achieved based on the expansion of the retinal image and depth information. The spatial location of the stimulus is encoded by a population of neurons. The direction of the looming stimulus is computed by monitoring the shift of the peak of neuronal activity in this population. The signal encoding the stimulus location is gated by the direction-selective neurons onto a motor heading map, which specifies the escape direction. The selection of a proper action is achieved through competition among different groups of motor neurons. The model is based on the analysis of predator-avoidance in frog and toad but leads to a comparative analysis of mammalian visual systems.

Key Words

neural network; looming; avoidance; anuran; sensorimotor integration

Pages 263-319

A Hierarchical Network of Control Systems that Learn: Modeling Nervous System Function During Classical and Instrumental Conditioning

By A. Harry Klopf, James S. Morgan, and Scott E. Weaver

Abstract

A computational model of nervous system function during classical and instrumental conditioning is proposed. The model assumes the form of a hierarchical network of control systems. Each control system is capable of learning and is referred to as an associative control process (ACP). Learning systems consisting of ACP networks, employing the drive-reinforcement learning mechanism (Klopf, 1988) and engaging in real-time, closed-loop, goal-seeking interactions with environments, are capable of being classically and instrumentally conditioned, as demonstrated by means of computer simulations. In multiple-T mazes, the systems learn to chain responses that avoid punishment and that lead eventually to reward. The temporal order in which the responses are learned and extinguished during instrumental conditioning is consistent with that observed in animal learning. Also consistent with animal learning experimental evidence, the ACP network model accounts for a wide range of classical conditioning phenomena. ACP networks, at their current stage of development, are intended to model sensorimotor, limbic, and hypothalamic nervous system function, suggesting a relationship between classical and instrumental conditioning that extends Mowrer's (1956, 1960a/1973) two-factor theory of learning. In conjunction with consideration of limbic system and hypothalamic function, the role of emotion in natural intelligence is modeled and discussed. ACP networks constitute solutions to temporal and structural credit assignment problems, suggesting a theoretical approach for the synthesis of machine intelligence.

Key Words

computational neuroethology; learning; control theory; networks

Pages 321-352

A Hierarchical Network of Provably Optimal Learning Control Systems: Extensions of the Associative Control Process (ACP) Network

By Leemon C. Baird III and A. Harry Klopf

Abstract

An associative control process (ACP) network is a learning control system that can reproduce a variety of animal learning results from classical and instrumental conditioning experiments (Klopf, Morgan, & Weaver, 1993; see also the article, "A Hierarchical Network of Control Systems that Learn"). The ACP networks proposed and tested by Klopf, Morgan, and Weaver are not guaranteed, however, to learn optimal policies for maximizing reinforcement. Optimal behavior is guaranteed for a reinforcement learning system such as Q-learning (Watkins, 1989), but simple Q-learning is incapable of reproducing the animal learning results that ACP networks reproduce. We propose two new models that reproduce the animal learning results and are provably optimal. The first model, the modified ACP network, embodies the smallest number of changes necessary to the ACP network to guarantee that optimal policies will be learned while still reproducing the animal learning results. The second model, the single-layer ACP network, embodies the smallest number of changes necessary to Q-learning to guarantee that it reproduces the animal learning results while still learning optimal policies. We also propose a hierarchical network architecture within which several reinforcement learning systems (e.g., Q-learning systems, single-layer ACP networks, or any other learning controller) can be combined in a hierarchy. We implement the hierarchical network architecture by combining four of the single- layer ACP networks to form a controller for a standard inverted pendulum dynamic control problem. The hierarchical controller is shown to learn more reliably and more than an order of magnitude faster than either the single-layer ACP network or the Barto, Sutton, and Anderson (1983) learning controller for the benchmark problem.

Key Words

optimal control, learning, Q-learning, hierarchical control

Pages 353-385

Place Learning and the Dynamics of Spatial Navigation: A Neural Network Approach

By Nestor A. Schmajuk and Hugh T. Blair

Abstract

This study presents a real-time neural network capable of describing place learning and the dynamics of spatial navigation. The network incorporates detectors that can be tuned to the values of visual angles of different landmarks as perceived from the spatial location where a reinforcing event is encountered. After a detector has been tuned its output generates an effective stimulus that peaks at the distance from the landmark where positive or negative reinforcement was encountered previously. The outputs of the tuned detectors become associated with the reinforcing event. The network generates spatial generalization surfaces that can guide navigation from any location that is within view of familiar landmark cues, even if that location has never been visited before. Spatial navigation is accomplished by adopting a stimulus-approach principle--that is, by approaching appetitive places and avoiding aversive places. When generalization surfaces are assumed to represent forces exerted by the animal, the dynamics of spatial movements can be described. Computer simulations were carried out for appetitive, aversive, and aversive- appetitive place learning. This article shows that the network correctly describes the navigational trajectories and dynamics of many spatial learning tasks.

Key Words

place learning; spatial mapping; cognitive mapping; neural networks

back to TOC, back to top

Adaptive Behavior, 1 (3)

Adaptive Behavior

Volume 1, Number 3

Winter 1993

Table of Contents

Jim-Shih Liaw and Michael A. Arbib

Neural Mechanisms Underlying Direction-Selective Avoidance Behavior