In mathematics, Jensen's inequality, named after the Danish mathematician Johan Jensen, relates the value of a convex function of an integral to the integral of the convex function. It was proved by Jensen in 1906^[1]. Given its generality, the inequality appears in many forms depending on the context, some of which are presented below. Euclid, Greek mathematician, 3rd century BC, known today as the father of geometry; shown here in a detail of The School of Athens by Raphael. ... Johan Ludwig William Valdemar Jensen, mostly known as Johan Jensen, (May 8, 1859—March 5, 1925) was a Danish mathematician and engineer. ... In mathematics, convex function is a real-valued function f defined on an interval (or on any convex subset C of some vector space), if for any two points x and y in its domain C and any t in [0,1], we have Convex function on an interval. ... In calculus, the integral of a function is an extension of the concept of a sum. ... 1906 (MCMVI) was a common year starting on Monday (see link for calendar). ...

Statements

The classical form of Jensen's inequality involves several numbers and weights. The inequality can be stated quite generally using measure theory, and can be further generalized to its full strength in a probabilistic setting. In mathematics, a measure is a function that assigns a number, e. ...

Finite form

For a real convex function φ, numbers x_i in its domain, and positive weights a_i, Jensen's inequality can be stated as:

and the inequality is clearly reversed if φ is concave. In calculus, a differentiable function f is convex on an interval if its derivative function f ′ is increasing on that interval: a convex function has an increasing slope. ...

As a particular case, if the weights a_i are all equal to unity, then

For instance, the log(x) function is concave, so substituting in the previous formula, this establishes the (logarithm of) the familiar arithmetic mean-geometric mean inequality: In mathematics, the inequality of arithmetic and geometric means, or more briefly the AM-GM inequality, states that the arithmetic mean of a list of non-negative real numbers is greater than or equal to the geometric mean of the same list; and further, that the two means are equal...

The variable x may, if required, be a function of another variable (or set of variables) t, so that x_i = g(t_i). All of this carries directly over to the general continuous case: the weights a_i are replaced by a non-negative integrable function f(x), such as a probability distribution, for example; and the summations replaced by integrals.

In measure-theoretic notation

Let (Ω,A,μ) be a measure space, such that μ(Ω) = 1. If g is a real-valued function that is μ-integrable, and if φ is a measurable convex function on the real axis, then: In mathematics, a measure is a function that assigns a number, e. ... In mathematics, the real numbers may be described informally in several different ways. ... The integral can be interpreted as the area under a curve. ... In mathematics, measurable functions are well-behaved functions between measurable spaces. ... In mathematics, convex function is a real-valued function f defined on an interval (or on any convex subset C of some vector space), if for any two points x and y in its domain C and any t in [0,1], we have Convex function on an interval. ...

In probability-theory notation (real space)

The same result can be stated in a probability theory setting. Let be a probability space, X an integrable real-valued random variable and φ a measurable convex function. Then: Probability theory is the mathematical study of phenomena characterized by randomness or uncertainty. ... In mathematics, a probability space or probability measure is a set S, together with a σ-algebra X on S and a measure P on that σ-algebra such that P(S) = 1. ... In mathematics, the term integrable function refers to a function whose integral may be calculated. ... A random variable is a mathematical function that maps outcomes of random experiments to numbers. ...

In this probability setting, the measure μ is intended as a probability , the integral with respect to μ as an expected value , and the function g as a random variable X. In probability theory the expected value (or mathematical expectation) of a random variable is the sum of the probability of each possible outcome of the experiment multiplied by its payoff (value). Thus, it represents the average amount one expects as the outcome of the random trial when identical odds are... A random variable is a mathematical function that maps outcomes of random experiments to numbers. ...

In probability-theory notation (general)

More generally, let T be a real topological vector space, and X a T-valued integrable random variable. In this general setting, integrable means that for any element z in the dual space of T: , and there exists an element in T, such that . Then, for any measurable convex function φ and any sub-σ-algebra of : In mathematics a topological vector space is one of the basic structures investigated in functional analysis. ... In mathematics, the term integrable function refers to a function whose integral may be calculated. ... In mathematics, the existence of a dual vector space reflects in an abstract way the relationship between row vectors (1×n) and column vectors (n×1). ... In mathematics, a σ-algebra (pronounced sigma-algebra) or σ-field over a set X is a collection Σ of subsets of X that is closed under countable set operations; σ-algebras are mainly used in order to define measures on X. The concept is important in mathematical analysis and probability theory. ...

Here stands for the expectation conditioned to the σ-algebra . This general statement reduces to the previous ones when the topological vector space T is the real axis, and is the trivial σ-algebra . In probability theory, a conditional expectation is the expected value of a real random variable with respect to a conditional probability distribution. ... ...

Proofs

A graphical "proof" of Jensen's inequality for the probabilistic case. The dashed curve along the X axis is the hypothetical distribution of X, while the dashed curve along the Y axis is the corresponding distribution of Y values. Note that the convex mapping Y(X) increasingly "stretches" the distribution for increasing values of X.

A proof of Jensen's inequality can be provided in several ways, and three different proofs corresponding to the different statements above will be offered. Before embarking on these mathematical derivations, however, it is worth analyzing an intuitive graphical argument based on the probabilistic case where X is a real number (see figure). Assuming a hypothetical distribution of X values, one can immediately identify the position of and its image in the graph. Noticing that for convex mappings the corresponding distribution of Y values is increasingly "stretched out" for increasing values of X, it is easy to see that the distribution of Y is broader than that of X in the interval corresponding to X > X₀ and narrower in X < X₀ for any X₀; in particular, this is also true for . Consequently, in this picture the expectation of Y will always shift upwards with respect to the position of , and this "proves" the inequality, i.e. Image File history File links Jensen_graph. ... Image File history File links Jensen_graph. ...

the equality taking place when is not strictly convex, e.g. when it is a straight line.

The proofs below formalize this intuitive notion.

Proof 1 (using the finite form)

If are two arbitrary positive real numbers such that λ₁ + λ₂ = 1, then convexity of implies for any . This can be easily generalized: if are n positive real numbers such that , then

for any . This finite form of the Jensen's inequality can be proved by induction: by convexity hypotheses, the statement is true for n = 2. Suppose it is true also for some n, one needs to prove it for n+1. At least one of the λ_i is strictly positive, say λ₁; therefore by convexity inequality: Mathematical induction is a method of mathematical proof typically used to establish that a given statement is true of all natural numbers. ...

Since , one can apply the induction hypotheses to the last term in the previous formula to obtain the result, namely the finite form of the Jensen's inequality.

In order to obtain the general inequality from this finite form, one needs to use a density argument. The finite form can be re-written as:

where μ_n is a measure given by an arbitrary convex combination of Dirac deltas: A convex combination is a linear combination of data points (which can be vectors or scalars) where all coefficients are positive and sum up to 1. ... The Dirac delta function, introduced by Paul Dirac, can be informally thought of as a function &#948;(x) that has the value of infinity for x = 0, the value zero elsewhere, and a total integral of one. ...

Since convex functions are continuous, and since convex combinations of Dirac deltas are weakly dense in the set of probability measures (as could be easily verified), the general statement is obtained simply by a limiting procedure. In mathematics, a continuous function is a function for which, intuitively, small changes in the input result in small changes in the output. ... In mathematics, weak topology is an alternative term for initial topology. ... In topology and related areas of mathematics, a subset A of a topological space X is called dense (in X) if, intuitively, any point in X can be well-approximated by points in A. Formally, A is dense in X if for any point x in X, any neighborhood of...

Proof 2 (measure theoretic notation)

Let g be a real-valued μ-integrable function on a measure space Ω, and let φ be a convex function on the real numbers. Define the right-handed derivative of φ at x as

Since φ is convex, the quotient of the right-hand side is decreasing when t approaches 0 from the right, and bounded below by any term of the form

where t < 0, and therefore, the limit does always exist.

Now, let us define the following:

Then for all x, . To see that, take x>x₀, and define t = x − x₀ > 0. Then,

Therefore,

as desired. The case for x < x₀ is proven similarly, and clearly .

φ(x₀) can then be rewritten as

But since μ(Ω) = 1, then for every real number k we have

In particular,

Proof 3 (general inequality in probabilistic notation)

Let X be an integrable random variable that takes value in a real topological vector space T. Since is convex, for any , the quantity

is decreasing as θ approaches 0. In particular, it is well defined the subdifferential of evaluated at x in the direction y, defined by:

It is easily seen that the subdifferential is linear in y and, since the infimum taken in the right-hand side of the previous formula is smaller than the value of the same term for θ = 1, one gets:

In particular, for an arbitrary sub-σ-algebra we can evaluate the last inequality when to obtain:

Now, if we take the expectation conditioned to on both sides of the previous expression, we get the result since:

by the linearity of the subdifferential in the y variable, and well-known properties of the conditional expectation. In probability theory, a conditional expectation is the expected value of a real random variable with respect to a conditional probability distribution. ...

Applications and special cases

Form involving a probability density function

Suppose Ω is a measurable subset of the real line and f(x) is a non-negative function such that

In probabilistic language, f is a probability density function. In mathematics, a probability density function (pdf) serves to represent a probability distribution in terms of integrals. ...

Then Jensen's inequality becomes the following statement about convex integrals:

If g is any real-valued measurable function and φ is convex over the range of g, then

If g(x) = x, then this form of the inequality reduces to a commonly used special case:

Alternative finite form

If Ω is some finite set , and if μ is a counting measure on Ω, then the general form reduces to a statement about sums: In mathematics, the counting measure is an intuitive way to put a measure on any set: the size of a subset is taken to be the number of the subsets elements if this is finite, and ∞ if the subset is infinite. ...

provided that

There is also an infinite discrete form.

Statistical physics

Jensen's equality is of particular importance in statistical physics when the convex function is an exponential, giving:

where angle brackets denote expected values with respect to some probability distribution in the random variable X. In probability theory the expected value (or mathematical expectation) of a random variable is the sum of the probability of each possible outcome of the experiment multiplied by its payoff (value). Thus, it represents the average amount one expects as the outcome of the random trial when identical odds are... In mathematics and statistics, a probability distribution, more properly called a probability density, assigns to every interval of the real numbers a probability, so that the probability axioms are satisfied. ... A random variable is a mathematical function that maps outcomes of random experiments to numbers. ...

The proof in this case is very simple (cf. Chandler, Sec. 5.5). The desired inequality follows directly, by writing

and then applying the inequality

to the final exponential.

Information theory

If p(x) is the true probability distribution for x, and q(x) is another distribution, then applying Jensen's inequality for the random variable Y(x) = q(x)/p(x) and the function φ(y) = −log(y) gives

a result called Gibbs' inequality. In information theory, Gibbs inequality is a statement about the mathematical entropy of a discrete probability distribution. ...

It shows that the average message length is minimised when codes are assigned on the basis of the true probabilities p rather than any other distribution q. The quantity that is greater than zero is called the Kullback-Leibler distance of q from p. In probability theory and information theory, the Kullback-Leibler divergence, or relative entropy, is a quantity which measures the difference between two probability distributions. ...

Rao-Blackwell theorem

Main article: Rao-Blackwell theorem

If L is a convex function, then from Jensen's inequality we get In statistics, the Rao-Blackwell theorem describes a technique that can transform an absurdly crude estimator into an estimator that is optimal by the mean-squared-error criterion or any of a variety of similar criteria. ...

So if δ(X) is some estimator of an unobserved parameter θ given a vector of observables X; and if T(X) is a sufficient statistic for θ; then an improved estimator, in the sense of having a smaller expected loss L, can be obtained by calculating In statistics, an estimator is a function of the observable sample data that is used to estimate an unknown population parameter; an estimate is the result from the actual application of the function to a particular set of data. ... In statistics, one often considers a family of probability distributions for a random variable X (and X is often a vector whose components are scalar-valued random variables, frequently independent) parameterized by a scalar- or vector-valued parameter, which let us call &#952;. A quantity T(X) that depends on...

the expectated value of δ with respect to θ, taken over all possible vectors of observations X compatible with the same value of T(X) as that observed.

This result is known as the Rao-Blackwell theorem. In statistics, the Rao-Blackwell theorem describes a technique that can transform an absurdly crude estimator into an estimator that is optimal by the mean-squared-error criterion or any of a variety of similar criteria. ...

See also

• Flaw of averages

There are very few or no other articles that link to this one. ...

References

• Walter Rudin (1987). Real and Complex Analysis. McGraw-Hill. ISBN 0-07-054234-1.
• David Chandler (1987). Introduction to Modern Statistical Mechanics. Oxford. ISBN 0-19-504277-8.
• Jensen, Johan Ludwig William Valdemar (1906). "Sur les fonctions convexes et les inégalités entre les valeurs moyennes". Acta Mathematica 30: 175-193.

Johan Ludwig William Valdemar Jensen, mostly known as Johan Jensen, (May 8, 1859&#8212;March 5, 1925) was a Danish mathematician and engineer. ... Acta Mathematica is a journal publishing original research papers in all fields of mathematics. ...

External links

• Jensen's inequality on MathWorld
• Jensen's inequality serves as the logo for the Mathematics department of Copenhagen University

Footnotes