A Bellman equation (also known as a dynamic programming equation), named after its discoverer, Richard Bellman, is a necessary condition for optimality associated with the mathematical optimization method known as dynamic programming. Almost any problem which can be solved using optimal control theory can also be solved by analyzing the appropriate Bellman equation. The Bellman equation was first applied to engineering control theory and to other topics in applied mathematics, and subsequently became an important tool in economic theory. Richard Ernest Bellman (1920–1984) was an applied mathematician, celebrated for his invention of dynamic programming in 1953, and important contributions in other fields of mathematics. ... In mathematics and computer science, dynamic programming is a method of solving problems exhibiting the properties of overlapping subproblems and optimal substructure (described below) that takes much less time than naive methods. ... Optimal control theory is a mathematical field that is concerned with control policies that can be deduced using optimization algorithms. ... In engineering and mathematics, control theory deals with the behavior of dynamical systems. ... Economics is the social science studying production and consumption through measurable variables. ...

The Hamilton-Jacobi equation of classical mechanics is an example of a Bellman equation. The Hamilton-Jacobi-Bellman (HJB) equation is a partial differential equation which is central to optimal control theory. ... Classical mechanics (also called Newtonian mechanics) is used for describing the motion of macroscopic objects, from projectiles to parts of machinery, as well as astronomical objects, such as spacecraft, planets, stars, and galaxies. ...

Principle of optimality

Given an appropriate initial condition

the canonical infinite horizon dynamic programming problem is:

subject to the constraints

In this problem, x is a vector of state and control variables, indexed by discrete time t. 0≤β≤1 is the discount factor. In finance, discounting is the process of finding the current value of an amount of cash at some future date, and along with compounding cash form the basis of time value of money calculations. ...

The recursive restatement of this problem as a Bellman equation is:

.

The function V that solves the Bellman equation is called the value function. The value function describes the optimized value of the problem, as a function of the state variable x. The function y(x) that describes the optimal choice as a function of the state is called the policy function.

Stokey & Lucas (1989: 67-77) called the equivalence between these two forms of the problem the principle of optimality. The principle asserts that if the policy function is optimal for the infinite summation, then it must be the case that whatever the initial state and decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from that first decision (as expressed by the Bellman equation). The principle of optimality is related to the concept of optimal substructure, and problems that exhibit optimal substructure can often be solved with dynamic programming. Figure 1. ...

Example

In reinforcement learning, a Bellman equation refers to a recursion for expected rewards. For example, the expected reward for being in a particular state s and following some fixed policy π has the Bellman equation: Reinforcement learning refers to a class of problems in machine learning which postulate an agent exploring an environment in which the agent perceives its current state and takes actions. ...

This equation describes the expected reward for taking the action prescribed by some policy π.

The equation for the optimal policy is referred to as the Bellman optimality equation:

It describes the reward for taking the action giving the highest expected return.

Applications in Economics

The first known economic application of a Bellman equation is Merton's seminal 1973 article on the intertemporal capital asset pricing model.^[1] The solution to Merton's theoretical model, one in which investors chose between income today and future income or capital gains, is a form of Bellman's equation. Because economic applications of dynamic programming usually result in a Bellman equation that is a difference equation, economists refer to dynamic programming as a "recursive method." The Intertemporal Capital Asset Pricing Model, or ICAPM, is a linear factor model with wealth and state variable that forecast changes in the distribution of future returns or income. ... In mathematics, a recurrence relation, also known as a difference equation, is an equation which defines a sequence recursively: each term of the sequence is defined as a function of the preceding terms. ...

Lucas & Stokey describes stochastic and nonstochastic dynamic programming in considerable detail, giving many examples of how to employ dynamic programming to solve problems in economic theory.^[2] This book led to dynamic programming being employed to solve a wide range of theoretical problems in economics, including optimal economic growth, resource extraction, principal-agent problems, public finance, business investment, asset pricing, factor supply, and industrial organization. Ljungqvist & Sargent apply dynamic programming to study a variety of theoretical questions in monetary policy, fiscal policy, taxation, economic growth, search theory, and labor economics.^[3] Dixit & Pindyck showed the value of the method for thinking about capital budgeting.^[4] Patrick L. Anderson used dynamic programming to develop methods to value closely held firms.^[5] World GDP/capita changed very little for most of human history before the industrial revolution. ... The related terms resource extraction and Resource extraction industry both refer to the practice of locating, acquiring and selling any resource, but typically a natural resource. ... In economics, the principal-agent problem treats the difficulties that arise under conditions of incomplete and asymmetric information when a principal hires an agent. ... Public finance (government finance) is the field of economics that deals with budgeting the revenues and expenditures of a public sector entity, usually government. ... Invest redirects here. ... Valuation is the process of estimating the value of an asset or liability. ... Classical economics distinguishes between three factors of production which are used in the production of goods: Land or natural resources - naturally-occurring goods such as soil and minerals. ... Industrial organization is the field of economics that studies the behavior of firms, the structure of markets and of their interactions. ... It has been suggested that monetary theory be merged into this article or section. ... Fiscal policy is the economic term that defines the set of principles and decisions of a government in setting the level of public expenditure and how that expenditure is funded. ... World GDP/capita changed very little for most of human history before the industrial revolution. ... In economics, search theory (or just search) is the study of an individuals optimal strategy when facing a series of potential opportunities of random quality and a cost of delaying choice. ... Labour economics seeks to understand the functioning of the market for labour. ... The process of determining which potential long-term projects are worth undertaking, by comparing their expected discounted cash flows with their internal rates of return. ... For other persons named Patrick Anderson, see Patrick Anderson (disambiguation). ...

Using dynamic programming to solve concrete problems is complicated by informational difficulties, such as choosing the unobservable discount rate. There are also computational issues, the main one being the curse of dimensionality arising from the vast number of possible actions and potential state variables that must be considered before an optimal strategy can be selected. For an extensive discussion of computational issues, see Miranda & Fackler.^[6] Curse of dimensionality is a term coined by Richard Bellman applied to the problem caused by the rapid increase in volume associated with adding extra dimensions to a (mathematical) space. ...

See also

• Hamilton-Jacobi-Bellman equation.
• optimal control theory

The Hamilton-Jacobi-Bellman (HJB) equation is a partial differential equation which is central to optimal control theory. ... Optimal control theory is a mathematical field that is concerned with control policies that can be deduced using optimization algorithms. ...

References

1. ^ Robert C. Merton, 1973, "An Intertemporal Capital Asset Pricing Model," Econometrica 41: 867-887.
2. ^ *Nancy Stokey, and Robert E. Lucas, with Edward Prescott, 1989. Recursive Methods in Economic Dynamics. Harvard Univ. Press.
3. ^ Lars Ljungqvist & Thomas Sargent, 2004. Recursive Macroeconomic Theory. MIT Press.
4. ^ Avinash Dixit & Robert Pindyck, 1994. Investment Under Uncertainty. Princeton Univ. Press.
5. ^ Patrick L. Anderson, 2004. Business Economics and Finance. CRC Press.
6. ^ Miranda, M., & Fackler, P., 2002. Applied Computational Economics and Finance. MIT Press.