In the previous two chapters, we considered how the brain forms maps of the environment as the body moves in it. Now and in the following chapters, we will discuss how the brain forms maps of the body in relation to the external world. As we move and act on the environment, our brain is like a crane operator who must act on a myriad of levers for commanding the motions of the limbs. Even the simplest actions, like reaching for a glass of water, require the coordinated control of some thirty-five arm and hand muscles. These muscles are divided in smaller compartments, each compartment operated by a single motor neuron. In the end there are thousands of little levers that our crane operator — our body — must act upon successfully for conducting the simple gestures. The operator receives information about the state of motion of each limb and about the forces applied by the muscles. There are two geometries for this scenario: the geometry of the body (of its motors and sensors), and the geometry of the world outside. We already encountered this duality in the navigation problem, where the geometry of the images projected on the eyes is different from the geometry of the surrounding space. Here, as well as in the navigation, what matters is that the two geometries are mutually connected.

3.1 Geometry vs. Dynamics

Consider the situation depicted in figure 3.1A. It describes an experiment carried out by Pietro Morasso (1981) while he was visiting the laboratory of Emilio Bizzi at MIT. The sketch is the top view of a subject holding the handle of a planar mechanism that was designed by Neville Hogan to study the reaching movements of the hand. We can describe the subject’s geometry in two alternative ways. We can focus on his hand, or we can focus on his arm. In the first case, the position of the hand on the plane is determined by the two coordinates (x,y), of its center, taken with respect to a Cartesian system. An alternative way (not the only one) is to look at the two joint angles: the shoulder angle, ϕ, and the elbow angle, θ. These different points of view are related to the different perspective that one may have. Earlier (p.68) motor physiologists, before Morasso’s time, focused on the movements of individual body segments. And, most often, of one body segment at a time, for example the upper arm motions about the elbow. On the other hand, robotics and artificial intel-ligence placed the focus on the goals of behaviors, for example, moving objects. Morasso came from the school of robotics and artificial intelligence. In his experiments, he paid attention to the motion of the hand.

In the planar configuration of figure 3.1A, the hand coordinates (x,y) and the joint coordinates (ϕ,θ) are connected by simple trigonometry:

where l₁ and l₂ are the lengths of the two arm segments. What is the geometrical interpretation of this relation between angular coordinates of the joints and Cartesian coordinates of the hand? First, it is a nonlinear relation. Suppose that we need to move the arm from an initial configuration (ϕ_I,θ_I) to a final configuration (ϕ_F,θ_F) of joint angles. A simple way to do so is by setting

Here, Δϕ = ϕ_F −ϕ_I, Δθ = θ_F −θ_I and u(t) is an arbitrary function of time that varies from 0 to 1 between the start and the end of the movement. In other words, we may drive the two joint synchronously from start to end locations. In this case it is easy to see—by eliminating u(t)—that the two angles are linearly related to each other:

We can apply the same logic to the movements of the hand in Cartesian coordinates and derive a rectilinear motion from (x_I,y_I) to (x_F,y_F).

The two rectilinear motions are not generally compatible. This is illustrated in figure 3.1B, where we see that straight lines in hand coordinates correspond to curved lines in joint coordinates, and vice versa. The diagram in this figure, however, is somewhat misleading. We know that it is appropriate to use Cartesian axes to describe the ordinary space in which the hand moves. We have seen in chapter 2 that Cartesian coordinates capture the metric properties of Euclidean space and, in particular, the independent concepts of length and angle. If we have two points P, Q, their Cartesian coordinates may change, depending on the origin and on the (p.69)

(p.70) orientation of the coordinate axes. But the distance computed with the sum-of-squares rule remains invariant.

What can we say about using Cartesian coordinates for the joint angles, ϕ and θ? If angles are represented by real numbers what prevents us from placing these numbers on two lines and call them Cartesian coordinates of an “angle space” ? In principle, this could be done (and is routinely done by scientists). However, by doing so, one neglects the critical fact that 0, 2 π, 4π, and so on are not really different angles. The numbers are different, but the angles are not. Because of their cyclical character, angular variables are better represented over circles than over lines. Angles have curvature! But then how do we represent two angular variables, as in our case of shoulder and elbow joint coordinates? As shown in figure 3.2, the natural geometrical structure for describing a two-link planar mechanism, like a double pendulum or our simplified arm, is a doughnut-shaped object known as a torus.

A torus, like a sphere is a two-dimensional geometrical object. A fundamental theorem by John Nash (1956) establishes that this type of objects can be placed inside an Euclidean space of higher dimension by an operation called an isometric embed-ding.¹ By this operation, we can represent the two angles of the arm in figure 3.1A as coordinates over a torus inside a three-dimensional Euclidean space (figure 3.2).

If we take two points, A and B, there is a unique line on the torus that has the smallest length. This is called a geodesic line. The linear joint interpolation of equation (3.2) defines such a geodesic line. Since the concept of a manifold is a generalization of the concept of a surface, the Euclidean plane is also a manifold, whose geodesics are straight lines. Summing up, the Cartesian coordinates of the hand (x,y)

(p.71) and the angular coordinates (ϕ,θ) describe two geometrical domains with different geodesic properties. If we move the hand between two points over a plane, what type of geodesic do we tend to follow? Morasso addressed this question by asking his subject to move his or her hand between targets on the plane. He found that movements tend to occur along straight lines — that is, along Euclidean geodesics (figure 3.3). Tamar Flash and Neville Hogan (1985) interpreted this finding as evidence that the nervous system seeks to optimize the smoothness of hand movements by moving along trajectories that minimize the integrated amplitude of the third time derivative of hand position, or “jerk”:

The idea that there is a separation between the geometry of movement — the “kinematics” — and its dynamical underpinnings did not go unchallenged and is still an object of controversy. Yoji Uno, Mitsuo Kawato, and Ryoji Suzuki (1989) suggested that, instead of optimizing the smoothness of hand motions, the nervous system was concerned (p.72) with minimizing variables that are more directly connected with effort. Then, they considered, as cost function, the integral of the net torque changes over each movement:

The term τ_i represents the joint torque produced by one of the n arm muscles. One can intuitively see the similarity between the two formulations. Consider a point mass. By Newton’s law, the point mass accelerates in proportion to the applied force. If one takes one more time derivative, one obtains a relation between the jerk and the rate of change of the force. This explains why the two optimization criteria yield relatively similar predictions of arm trajectories. There is however an important difference between kinematic and dynamic criteria. The outcome of the dynamical optimization depends on properties such as the inertia of the moving arm. The minimization of the cost integral (3.5) requires knowing the relation between forces and motions, as expressed by the limb’s dynamical equations. These equations are quite complex and grow in complexity as more degrees of freedom are introduced. The reader can find explicit forms for two-joint dynamics in standard robotics hand-books. Here, we do not want to get into excessive detail and we limit ourselves the describe dynamics as a second-order nonlinear ordinary differential equation (ODE), whose general form² is

The generalized coordinate q = [ϕ,θ]^T is the configuration of the arm and the corresponding generalized force is a joint-torque vector Q = [τ_ϕ,τ_θ]^T. Thus, with the vector notation of equation (3.6) we are representing a system of two coupled nonlinear ODEs. The optimization by Uno and colleagues requires us to calculate the torque vector based on these ODEs. On the other hand, the kinematic approach of Flash and Hogan is strictly geometrical and does not involve forces (that is, dynamics). Unlike dynamic criteria, the geometrical optimization predicts that neuromuscular activities are coordinated by the brain to generate a rectilinear motion of the hand or of what-ever other element plays the role of a controlled “endpoint element,” and this result will not be affected by changes in the dynamical structure controlled by the muscles.

Randy Flanagan and Ashwini Rao (1995) performed a simple and elegant variant of Morasso’s experiment. They asked a group of volunteers to control a cursor by moving the hand over a plane. However, in this case, the position of the cursor was not an image of the position of a subject’s hand but was in direct relation to his or her shoulder and elbow angle. This task was difficult and somewhat confusing for the subjects. However, after some practice, they learned to move the cursor onto the targets. Most remarkably, as they learned to do so, they gradually but consistently and spontaneously learned to produce rectilinear movements of the cursor, at the (p.73)

expenses of more curved motions of the hand (figure 3.4).This result is incompatible with the notion that the nervous system attempts to minimize a dynamic criterion, such as the minimum-torque change of equation (3.5).

3.2 Does the Brain Compute Dynamics Equations?

Until the late 1970s, the study of neurobiology was mostly limited to the control of single muscles or of muscle pairs acting on a single joint. A large volume of work was dedicated to the control of eye movements, where six extrinsic muscles are responsible to move the eye and to control the direction of the gaze. The engineering frame-works for oculomotor control and for the control of limb movements derived from (p.74)

the theory of linear control systems and of feedback control in particular. Computational models were abundant in “box diagrams” such as the one in figure 3.5. Robotics research and, in particular, the theory of robot manipulators brought to the neurobiology of motor control a new awareness of the dynamical complexity of multiarticular structures such as the human arm. Let us compare the inertial dynamics of the upper arm about the elbow joint and the inertial dynamics of the two-joint arm of figure 3.1. The first are described by the following equation:

This is a straightforward translation of ma = F in angular terms. Here, I is the moment of inertia of the forearm about the axis of rotation of the elbow, q̈ is the angular acceleration about the same axis, and Q is the corresponding joint torque vector. In the two-joint case, we have two angles that we call q₁ and q₂ (instead of ϕ and θ, for notational convenience) and two corresponding torques, Q₁ and Q₂.The inertial dynamics look like this:

(p.75) The reader who is not acquainted with this equation can find its derivation at www.shadmehrlab.org/book/dynamics.pdf. Here, we wish to focus the attention not on the detailed form of these equations but on a few relevant features. First, note how the transition from equation (3.7) to equation (3.8) has led to a disproportionate increase in size. In the first we have a single multiplication, whereas in the second one we count 15 additions, 42 multiplications, and a couple of trigonometric functions. It is not a pretty sight. If we were to write the detailed equations for the entire arm in 3D, we would fill a few pages of text. To put some logical order in these equations, we may start by noticing that some terms contain accelerations and some contain velocities. The terms that contain acceleration are the first two in each equation. They may be written in vector-matrix form as the left end term of equation (3.7), that is,

This is called the inertial term. The acceleration appears linearly, as it is typical of all mechanical systems derived from Newton’s equation. However, the inertia matrix I depends on arm configuration, unlike the inertial term of equation (3.7), which is constant. In addition to this term, the two-link arm equation also contains terms that depend on the angular velocities. These are the “centripetal” terms, depending on the squares of the velocities and the “Coriolis” term depending on the product of the velocities. We may collect these terms in a single vector G(q, q̇) and the dynamical equation assumes the much less intimidating form

While this simple algebraic representation provides a valid description for more complex structures, such as a typical robotic manipulator or the whole human arm, it highlights the most relevant difference between the dynamics of a single joint and the dynamics of multiarticular limb and points to some computational challenges. One such challenge stems from the dependence of the torque on one joint upon the state of motion of another. These dynamics cannot be simplified by treating each dimension independently of the others. How important are these cross-dependencies in natural arm movements? Equation (3.8) provides us with a simple and direct way to answer this question. It is sufficient to record the trajectory of a reaching arm movement, derive the corresponding angular motion of the shoulder and elbow joints, take the first and second derivatives, and plug these data into the equations. John Hollerbach and Tamar Flash (1982) performed this experiment, using the same apparatus and task of Morasso. When they calculated the different contributions to the net torques at the shoulder and elbow, they were able to conclude that the (p.76)

interaction components that are summarized by the nonlinear term G(q, q̇) are quite substantial compared to the pure inertial terms. As an example, consider the trajectory in figure 3.6A and B, which is similar to the movement labeled as “e” in figure 3.3. The movement time is about half a second, corresponding to a relatively rapid but not unusual speed. The different components of the torque profiles at the shoulder and elbow joint are plotted in figure 3.6C and D, respectively. Note the amount of toque that is due to centripetal and Coriolis terms, and also the amount of inertial torque at the shoulder due to the acceleration at the elbow and vice versa. Most important, the same motion of the hand between two different targets would lead to qualitatively different torque profiles. This evidence was sufficient to conclude that arm movements cannot be controlled by some linear feedback mechanism, such as the one depicted in figure 3.5 for the oculomotor system.

(p.77) 3.3 The Engineering Approach

Let us briefly switch our perspective from neuroscience to robotics. Control engineers are primarily concerned with the concept of stability. Loosely speaking, a system is stable if, given some unexpected perturbation it will eventually go back to its unperturbed state. However, “eventually” is not strong enough a requirement for practical applications. A stronger form of stability requires the system to converge exponentially in time to the unperturbed state. One of the objectives, if not the main objective, of feedback control is to insure stability against uncertainties that arise from limited knowledge of the environment and of the controlled “plant.” Importantly, robotic engineers have some good prior knowledge of their own devices. So, for example, they would know to good accuracy the parameters in equation (3.9). Given this knowledge, we now want to write the right-hand term, the torque vector, as the driving control input to the arm. Because we are dealing with feedback control, we think of Q as a function of the state of motion—that is, the vector [q,q̇]^T—and of time. A physical system whose dynamics equations do not depend upon time is said to be autonomous. For such system the future is entirely determined by a measure of the state at some initial instant. For example, the equation of a simple oscillator, like a mass attached to a spring, is

The solution of this equation is

The phase and the amplitude are determined by the initial position q₀=q(t₀) and velocity v₀=q(t₀). Therefore, by setting (or measuring) these initial conditions, we know how the oscillator will behave for all future times. However, if we add an external input, as in

then the trajectory of the system is no longer determined by the initial state alone and depends upon the future values of u(t) as well.³

Now, consider the stability problem. Suppose that we have a robotic arm, governed by a feedback controller that attempts to track a desired trajectory, q̂(t).Then, the combined system + controller equation takes the form

Our goal is to design the control function Q(q, q̇, q̂(t)) so that the resulting movement is exponentially stable about the desired trajectory. As we stated before, a (p.78) robotics engineer would have a good model of the controlled mechanical arm. The model would contain the two terms

We can use these terms to design the control function as

Substituting this in equation (3.13) with the assumptions of equation (3.14) we get the simple double-integrator system:

It is important to observe that the symbol ≈ is not the ordinary equals sign, but an approximate equality. We will come back to this in a little while but for the next few lines we will use the standard equality. We begin by giving a particular form to the term a(q, q̇, t).Starting from the desired trajectory, we calculate the desired velocity, $\dot{\widehat{q}}\left(t\right)$ and acceleration, $\ddot{\widehat{q}}\left(t\right)$, by taking the first and second time derivatives of q̂(t) Then, we set

and

The last equation defines the tracking error as a function of the current position and time. The two matrices K_P and K_D contain gain coefficients that multiply the position and velocity errors.⁴ By combining equations (3.18), (3.17), and (3.16) we finally obtain a linear second-order differential equation for the error function:

From calculus we know that by setting

we can rewrite equation (3.19) as N decoupled equations (p.79)

whose general solution is

Therefore, with the appropriate choice of the two gain matrices, we can insure that the tracking error will go to zero in exponential time. This type of dynamic stability was observed in reaching movements by Justin Won and Neville Hogan (1995).They asked subjects to execute reaching movements of the hand while holding a planar manipulandum (figure 3.7). Most of the times, the manipulandum allowed for free motion of the hand in the plane, and the subjects moved along straight lines from start to end target. In some random trials, one of the joints of the manipulandum was blocked at the onset of the movement by an electromagnetic brake. As a consequence, in these test trials the movement of the hand was initially constrained to remain on a circular arc (dotted line). Shortly after the start, the brake was released. The resulting motion of the hand converged rapidly to the unperturbed path, thus demonstrating a dynamic stability of the planned movement trajectory, similar to the exponential stability discussed earlier.

(p.80) But now remember that in equations (3.21) and (3.22), instead of = we should have used ≈ as the model of the arm dynamics that can at most approximate the actual value of the parameters. However, for small deviations, the uncertainty of the model can be regarded as an internal perturbation that does not critically compromise stability. The essential point is that if we have an internal representation of the dynamics, a control system can generate behaviors that are stable, in the face of uncertainties concerning the environment, and the controlled structure.

3.4 Does the Brain Represent Force?

The idea of model-based control was born as an engineering idea. What can it tell us about the brain? One of the fathers of computer technology, the mathematician John von Neumann, was driven by the desire to create an artificial brain. And some of the earliest computational neuroscientists, like Warren McCulloch and Walter Pitts, described neurons as logical gates. But it is hard to capture the intricacies of biology within the confines of engineering and mathematics. And the idea that the brain computes something like the dynamics equations for the limb it controls has encountered — and still encounters to this day — some strong resistance. One of the strongest counterarguments to the idea of explicit computations comes from the observation that ordinary physical systems behave in ways that may be described by equations but do not require solving equations with calculus-like rules. If you drop a stone, the stone will follow a simple and regular trajectory that you can derive by integrating the equations of motion in the gravitational field. Yet the stone does not compute integrals! In the same vein, an important concept in science is to seek for principles that can explain observations based on simple rules of causality.

Consider another example. Take the two opposing springs and dampers connected to a mass (figure 3.8A). These are physical elements that generate forces acting on the point mass. The two springs define a static equilibrium point. If an external agent places the point mass away from this equilibrium, the physical system will eventually bring it back there, because the equilibrium is stable. This behavior can be readily implemented by analog or digital elements (figure 3.8B) that produce forces in response to measured position and velocity of the point mass. Thus, we have a physical system and a computational system (which is nothing but another type of physical system!) that do the same job. In 1978, Andres Polit and Emilio Bizzi published a finding that physiologists of that time found hard to digest. They trained monkeys to reach a visual target by a movement of the upper arm about the elbow. A monkey could not see its arm as it was moving under an opaque surface and was rewarded when it pointed correctly to the visual target. Not surprisingly, that monkey learned the task. However, the monkeys were able to perform the task after they were surgically deprived of all sensory information coming from their arm. And they (p.81)

successfully reached the targets even if the arm were unexpectedly displaced by a motor before starting the movement. If one were to think of movement control according to the scheme drawn in figure 3.8B, this result would not make any sense, because now the lines carrying the information about position and velocity were cut! But the scheme of figure 3.8A is perfectly consistent with the finding of Polit and Bizzi: If the muscles act upon the arm, as opposing viscoelastic elements, then all what the nervous system has to do is to set their equilibrium point to the target. No need to monitor the position and state of the arm in order to correct for errors. Instead, in order to move a limb the brain can set the equilibrium point so as to follow a desired movement trajectory.

Anatol Feldman pioneered this idea in 1966.Then, as well as in more recent works, Feldman proposed that we move our bodies by setting the equilibrium postures specified by reflexes and muscle mechanical properties. Does this make dynamic computations unnecessary? Answering this question is not quite as simple as it seems.

One may ask if to move your arm your brain must represent forces, as defined by Newton. Consider the equation that represents the behavior of a spring-mass-damping system, such as the one shown in figure 3.8: (p.82)

We derive this by substituting the expression of the forces generated by the spring and damper in F=mẍ. As a result the equation does not contain force at all. It has only the state (position and velocity) its derivative (acceleration) and an external input, u(t), that sets instant by instant the equilibrium point of the springs. The situation is not different from the one described in the previous paragraph to derive the controller for a robotic arm. We may not need to represent forces to derive a trajectory produced by the input function, but we need to know how the system reacts to applied forces. These are the terms that appear in equation (3.23). This equation allows us to derive the trajectory x(t) that results from the commanded equilibrium-point trajectory u(t). And the same equation allows us to derive the equilibrium-point trajectory from the desired movement.The only condition for this to be possible is K ≠ 0. Then, given a desired trajectory x̂(t) with first and second time derivatives $\dot{\widehat{x}}\left(t\right)$ and $\ddot{\widehat{x}}\left(t\right)$ the commanded equilibrium point-trajectory is

This is not substantially different from what we have done in deriving the engineering controller of equation (3.17). The only new element here is the focus on representing the control signal as a moving equilibrium point that drives the con-trolled “plant” along the desired trajectory. Intuitively, if K is sufficiently large and if the motion is smooth and slow, the first two terms in equation (3.24) may be small enough to be ignored and then all what one needs to do is to move the equilibrium point along the desired trajectory.This is the attractive promise of equilibrium-point control: there is no need to compute dynamics. The mechanical properties of the body and, in particular, the viscoelastic properties—responsible for the K and B terms—of the neuromuscular system may provide a way to avoid complex computations altogether. Interestingly, this viewpoint emerged vigorously at the same time in which the robotics viewpoint on inverse dynamic computations was becoming influential in the neuroscience of the motor system. But is it true that dynamics computations may be avoided?

Consider the simple movement of the right arm illustrated schematically in figure 3.9. This is a simulation of a simplified model of the arm, based on equation (3.8). The numerical parameters are listed in the box. We rewrite the dynamics equations to include the viscoelastic terms and make it formally similar to equation (3.23):

An appealing idea is to simply move the equilibrium point along the desired trajec-tory of the hand. The panels on the left illustrate the outcome of this approach. If the movement is sufficiently slow, for example if one takes two seconds to move the (p.83)

equilibrium point about 40 cm across, the hand moves in a nearly rectilinear path. However the deviation from the desired trajectory increases as the planned motion becomes faster (0.8 sec) and when the movement is more proximal to the body (figure 3.9C).Then, we may ask what would it take to change the equilibrium trajectory, u(t), in such a way to obtain the desired straight motion of the hand. We can certainly move our hand in a nearly straight 40 cm line within less than one second. To find the equilibrium point trajectory that achieves this goal, we begin by transforming the desired rectilinear trajectory of the hand, r(t) = [x(t),y(t)]^T into the trajectories of the shoulder and elbow joints, q(t) = [q₁(t),q₂(t)]^T with their first and second time derivatives. Then, we solve equation (3.25) algebraically to obtain

(p.84) The results of this simulation are shown in the left panels, B and D for the distal and proximal arm configurations. Only a small correction of the equilibrium point trajectory is sufficient to “straighten” the movement at low speed (2 sec). However, as the movement becomes faster (0.8 sec) the equilibrium point must take a quite distorted path in order to keep the hand on track. It is evident that we could not derive this complex equilibrium trajectory by performing a simple linear manipulation on the errors — shown on the left panels — that we would make by moving the equilibrium on the straight path. Here, to derive the motion of the equilibrium point that produces a desired actual movement of the hand we have explicitly solved an inverse dynamics problem. Our brain is likely using some different method. But, as we are indeed capable to move our limbs with dexterity, this method must effectively solve the same inverse dynamics problem.

3.5 Adapting to Predictable Forces

So far, we have described the basic computational idea of an internal model. Is there a more stringent empirical basis for this idea? Perhaps the first empirical evidence came from an experiment that the authors of this book performed in the early 1990s, while they were in the laboratory of Emilio Bizzi at MIT. We asked volunteers to move their arm while holding the handle of a planar manipulandum (figure 3.10A). They performed reaching movements of the hand, as in Morasso’s experiments. The robot, designed by Neville Hogan, was a lightweight two-joint device, equipped with position sensors and with two torque motors. The motors were “back-drivable,” meaning that when they were turned off, they generated only minimal resistance to motion. And the manipulator itself was made of light aluminum bars, so that the whole thing did not impede appreciably the free motion of the hand.

Therefore, with the motors turned off, the hand trajectories were rectilinear and smooth (figure 3.10B). After acquiring unperturbed movement data, the motors were programmed to generate a force, F, that depended linearly upon the velocity of the hand, ṙ:

This force field was equivalent to a controlled change in the dynamics of the subject’s arm. During the initial phase of the experiment, the subject’s dynamics were as in equation (3.13). Without making any assumption about the control function, Q(q,q̇,t), that a subject used to perform a particular trajectory qA(t), the dynamics equations are reduced to an algebraic identity over this trajectory. If the force field (3.27) is introduced without changing the control function, then the new limb dynamics are (p.85)

(p.86)

where J(q) is the Jacobian of the kinematic map from joint angles to hand position in Cartesian coordinates.With r = [x,y]^T and q = [q₁,q₂]^T the Jacobian is a 2 ×2 matrix of partial derivatives:

The Jacobian is a function of the arm configuration (i.e., the joint angles) as the kinematic map is nonlinear. We derived equation (3.28) from the force field equation (3.27) and from the expression of the Jacobian by combining the transformation from joint to hand velocity,

with the transformation from hand force to joint torque

We can therefore regard the field as an unexpected change on the function G(q, q̇):

The solution of equation (3.28) is a new trajectory qB(t) ≠qA(t). Figure 3.10C shows the response of a subject to this sudden dynamic change. It is important to observe that the force field and, accordingly, the change to G vanishes when the hand is at rest. This has two consequences. First, the subjects could not know if the field is on or off before starting to move, and second, the field does not alter the static equilibrium at the end of each movement. Therefore, we were not surprised to see that the effect of the unexpected force field on the trajectories was to cause a hook-shaped deviation, with the hand landing eventually on the planned target. While further studies revealed the presence of online corrections, these are not needed to maintain the final position unchanged by the force field. In the second part of the experiment, subjects were continuously exposed to the force field, with only sporadic “catch trials” in which the field was unexpectedly suppressed.

Figure 3.11 shows the evolution of motor performance across four consecutive blocks of trials. The four trajectories on the left—F1 to F4—were produced by a subject moving in the force field. The trajectories on the right—C1 to C4—were obtained from the same subject in the same four periods, but when the force field was unexpectedly removed. These are the catch trials, where one can observe the aftereffect of learning, that is, how the motor command evolved as the subject (p.87)

learned to move inside the force field. If one compares the first set of movements in the field (F1 and figure 3.10D), to the last set of aftereffects (C4) one can immediately observe that the shape of the aftereffect at the end-of learning is qualitatively the mirror image of the initial effect of the exposure to the field. This is easily understood in the dynamical framework. As the subjects learn to move in the field, they compensate for the forces that are generated by it along the desired trajectory. We represent this process by adding a term to the original controller:

For this equation to admit the original trajectory as a solution it is sufficient that

along q_A(t).This is equivalent to perform a local approximation of the external field. In this case, if the external field is suddenly removed after learning, this approximation—that is, the internal model of the field—becomes a perturbation with the opposite sign: (p.88)

This, however, is not the only way for our motor system to compensate for the external force. We discuss briefly two alternatives: stiffen up, or create a motor tape. The first consists in increasing the rigidity of the moving arm by coactivating opposing muscles. This is what you would do if you want to resist movement when someone shakes your hand. You activate the biceps and the triceps simultaneously. The two muscles oppose each other and the result is that the forearm becomes more rigid. Our brains chose this approach whenever facing an environment that is hard or impossible to predict. For example when holding the rudder of a sailboat and you need to keep the heading as steady as possible. In our force-field experiment, the presence of aftereffects is sufficient to rule out this type of control response: subjects were not merely becoming stiffer but they learned to counteract forces that—by design—were not random but predictable. The second possibility, instead, would be consistent with our observation of aftereffects. Equation (3.34) does not imply that Δ must be a representation of the force field on the left-hand side. The applied forces were designed to depend of the state of motion of the.arm, [q^T,q̇^T]^T. However, to recover the initial trajectory it would be sufficient for Δ to be a pure function of time,

This is simply a “motor tape” that plays back the forces encountered along the desired trajectory. The motor tape does not contain any information about the dependence of the external force upon the state variables. A second experiment was needed to address this possibility.

Michael Conditt, Francesca Gandolfo, and Sandro Mussa-Ivaldi (1997) tested the hypothesis of a motor tape with a simple experiment. They trained a group of volunteers to make reaching movements in a force field. The duration of each movement was typically less than one second. Therefore, after learning to reach targets in the field, the motor tapes would have the typical duration of each movement and would be “specialized” to reproduce selectively the force sequences encountered in the training phase. The hypothesis then predicts that subjects would not be able to compensate for forces encountered while performing different movements, like drawing a circle, a triangle or figure-eight pattern. Even if these drawing movements take place in the same region of space and velocity ranges as the reaching movements, the states of motions are traversed in different temporal sequences. The drawing movements last at least twice as long as the reaching movements. Nevertheless, the subjects of this experiment, after training to reach in the force field, were perfectly capable to compensate for the disturbing forces while drawing circles and other figures. A second group of subjects, instead of practicing reaching movements, (p.89) practiced drawing circles in the force field. The drawing movements of these subjects after training were not distinguishable from the drawing movements of subjects in the first group, who trained with reaching. These findings are sufficient to rule out the idea that subjects learned a motor tape and provide instead additional support to the concept that the brain constructed, through learning a computational representation of the dynamics as a dependence of the actions to be produced by the control system—in this case the function Δ(q, q̇)—upon the state of motion of the limb. While the represented entity does not have to be a force in the Newtonian sense, the resulting behavior is consistent with the representation of movement dynamics as formulated by classical laws.

3.6 Another Type of State-Based Dynamics: Motor Learning

The force-field experiments demonstrate the keen ability of the brain to deal with dynamics and with the concept of “state.” In mechanics, the state is a minimal set of variables that is sufficient to predict the future evolution of a system. In a single motor act, such as reaching for an object, the state in question is the mechanical state of the limb. Because the laws of classical mechanics govern the movement, the state is a collection of position and velocity variables. There is another kind of state that is relevant to motor behavior. Learning experiments have demonstrated that it is possible to predict how knowledge and control evolve through time based on past experience and of incoming information. Learning itself can therefore be observed as a dynamical system. This perspective will be developed in the remainder of this book. Here, we briefly introduce some of the key concepts of this approach. Let us start again from Newton’s equation for a point mass, F= mẍ. We derive the velocity of the point mass by integrating once this equation:

Instead of looking at the velocity as a continuous function of time, we may take samples at discrete instants of time. Then, provided that these instants are separated by small time intervals, δt, we may use Newton’s equation to derive as an approximation the velocity at the next instant ẋ⁽ⁿ⁺¹⁾ from the velocity ẋ⁽ⁿ⁾ and the external force F⁽ⁿ⁾ at the current instant. This is the difference equation:

If the mechanical work in an infinitesimal displacement is an exact differential, then the force is the gradient of a potential energy function, (p.90)

We now can rewrite the difference equation for a point mass in a potential field as

with

Opher Donchin, Joseph Francis, and Reza Shadmehr (2003) proposed to describe motor learning with a differential equation that has a similar form. However, in this case, the state variable in question is not the velocity or the position of the arm, but the internal model of the force field in which the arm is moving. They represented the internal model as a collection of basis functions over the state space of the arm. For example, each basis function could be a Gaussian centered on a preferred velocity. But this is certainly not the only possible choice. Referring to equation (3.34), let us represent the force field model in generalized coordinates as

where g=[g₁(q, q̇),g₂(q, q̇),…, g_m(q, q̇)]^T is a collection of m scalar basis functions and W is a n ×m matrix of coefficients. For a two-joint arm, n = 2. But, more generally, n is the dimension of the configuration space or, equivalently, the number of degrees of freedom under consideration. Then, the matrix W can also be written as an array of m n-dimensional vectors, one for each basis function

and the basis functions are scalars that modulate the amplitude of the w vectors over regions of the workspace, in a way similar to the receptive field of a sensory neuron. As we perform a series of reaching movements in the force field, q⁽¹⁾(t),q⁽²⁾(t), …, q^(k)(t), … we experience a corresponding sequence of “force trajectories”:

Suppose that on the ith movement, we intended to move along the trajectory q _A(t). Then, based on the internal model that we had at that point, we expected to experience a force (p.91)

Here, we have that the variable portion of the internal model is the coefficient matrix W. We assume that the basis functions remain fixed through time. This, of course is for conceptual convenience. More complex (and also more realistic) models of learning would allow for variations in all terms of Δ. Most importantly, at the end of the ith movement we have a discrepancy—an error— between what we experienced and what we expected to experience. This is a function

If we integrate the Euclidean norm of this function along the movement, we obtain a scalar quantity, a real number

This is an overall error that we have experienced across the movement and this error depends on the parameters of the internal models. The purpose of learning is to get better at what we are doing. In this case it means to make the expectation error as small as possible. A straightforward way to do so is by changing the parameters of the internal model so as to minimize the error on the next trial. If we expand the argument of the integral (3.46), we see that it is a quadratic form of the parameters:

Therefore, the error function E⁽ⁱ⁾(W⁽ⁱ⁾) has a global minimum that we reach by changing W along the gradient:

This is a difference equation that tells us how the internal model of the field changes in time. We compare it with equation (3.40) that describes the motion of a point-mass in a potential field. The two equations have different order. Newtonian mechanics is of the second order while this simple learning model is of the first. Both equations tell us how a state variable changes in time under the action of an external input. In the case of the mass, the input is the mechanical force, which the gradient of a potential energy function. In the case of learning, the “driving force” is the gradient of the prediction error. Perhaps, the most important common feature of (p.92) Newtonian mechanics and the description of learning as a dynamical system is that both allow us to connect theory with experiment. This will be the leading thread of the chapters that follow.

Summary

Space maps are relevant not only to navigation but also to movements in general. Here, we consider the maps that our brains must form to manipulate the environment. The kinematics maps relate the motions of the hand to the motions of the arm and come in two forms: direct kinematics, from arm’s joint angles to hand position, and inverse kinematics, from hand position to arm’s joint angles. These maps are nonlinear and, as a consequence, a straight line in hand space maps into a curve in arm space and vice versa. This reflects the fact that a space of angular variables is inherently curved, while the Euclidean space where the hand moves is flat. Observation of human movements over a plane demonstrated that the hand tends to follow rectilinear trajectories.

Observations with altered kinematics and with force perturbations demonstrated that the kinematics of arm movement is not a collateral effect of the brain attempting to economize or simplify the generation of muscle forces. Instead, to produce naturally observed movements, the brain must effectively solve a complex problem of dynamics. The solution of this problem does not necessarily involve any explicit representation of Newtonian forces, but it requires the ability to relate the motor commands that reach the muscles to the consequences in terms of changes in state of motion of the arm. This connection is represented in the arm’s dynamics equations, which contain terms representing the inertial properties of the arm and terms representing the viscoelastic properties of the muscles. The motor commands need not be a force and may represent instead the static equilibrium point expressed by the neuromuscular system. This is the position at which the arm would eventually settle under the influence of opposing muscle forces. Controlling the motion of this equilibrium point for producing a desired movement of the hand is equivalent to solving an inverse dynamic problem.

The first experimental evidence for internal models of dynamics derived from observing how subjects learned to perform reaching movements in the presence of deterministic force fields. These fields apply to the hand a force that depends upon the hand’s state of motion, that is, its position and velocity. The observations were consistent with the subjects gradually developing the ability to predict and cancel the applied forces along the movement trajectories. The sudden removal of the field results in aftereffects of learning, that is, in a perturbation that mirrors the initial (p.93) effect of the unexpected application of the field. The aftereffects, as well as the pattern of learning generalizations, demonstrate that the subjects learn to predict the functional form of the perturbation, which effectively corresponds to a change in the controlled dynamical system.

We introduce the concept that motor learning is a dynamical process whose state is the associated internal model of the body and its environment. In its simplest form, learning to move in a force field is described by a first-order differential equation whose state variable is the model of the field and whose input is the gradient of the prediction errors. In this respect, movement and learning are two concurrent processes operating over different time scales. (p.94)