NationMaster - Encyclopedia: Cross entropy

FACTOID # 141: Norwegians drink 10.7 kilograms of coffee per person each year. They also lead the globe in anxiety disorders. Maybe it’s time to switch to herbal tea.

Interesting food facts »

Home

Encyclopedia

WHAT'S NEW

RECENT ARTICLES

More Recent Articles »

SEARCH ALL

FACTS & STATISTICS Advanced view

Search encyclopedia, statistics and forums:

(* = Graphable)

Encyclopedia > Cross entropy

In information theory, the cross entropy between two probability distributions measures the overall difference between the two distributions. Cross entropy is closely related to Kullback-Leibler divergence (which is also known as the relative entropy). Information theory is the mathematical theory of data communication and storage founded in 1948 by Claude E. Shannon. ... In mathematics, a probability distribution assigns to every interval of the real numbers a probability, so that the probability axioms are satisfied. ... In probability theory and information theory, the Kullback-Leibler divergence, or relative entropy, is a quantity which measures the difference between two probability distributions. ...

The cross entropy for two distributions $p$ and $q$ over the same probability space is defined as follows: In mathematics, a probability space or probability measure is a set S, together with a Ïƒ-algebra X on S and a measure P on that Ïƒ-algebra such that P(S) = 1. ...

where $H (p)$ is the entropy of $p$ and $K L$ is the Kullback-Leibler divergence. For other senses of the term entropy, see entropy (disambiguation). ...

For discrete $p$ and $q$ this means In mathematics, a random variable is discrete if its probability distribution is discrete; a discrete probability distribution is one that is fully characterized by a probability mass function. ...

$mathrm{H}(p, q) = -sum_x p(x), log q(x). !$

The situation for continuous distributions is analogous: By one convention, a random variable X is called continuous if its cumulative distribution function is continuous. ...

$-int_X p(x), log q(x), dx. !$

NB: The notation $H(p, q)$ is sometimes used for both the cross entropy as well as the joint entropy of $p$ and $q$ . The joint entropy is an entropy measure used in information theory. ...

When comparing a distribution $q$ against a fixed reference distribution $p$ , cross entropy and KL divergence are essentially the same concept. In fact, they are identical up to an additive constant (since $p$ is fixed): both take on their minimal values when $p = q$ , which is $0$ for KL divergence, and $H(p)$ for cross entropy.

See also GA_googleFillSlot("encyclopedia_square");

See also