The entropy rate of a stochastic process is, informally, the time density of the average information in a stochastic process. For stochastic processes with countable index, the entropy rate H(X) is the limit of the joint entropy of n members of the process X_k divided by n, as n tends to infinity: In the mathematics of probability, a stochastic process is a random function. ... In the mathematics of probability, a stochastic process can be thought of as a random function. ... In mathematics the term countable set is used to describe the size of a set, e. ...

when the limit exists. An alternative, related quantity is:

For strongly stationary stochastic processes, H(X) = H'(X).

Entropy Rates for Markov Chains

Since a stochastic process defined by a Markov chain which is irreducible and aperiodic has a stationary distribution, the entropy rate is independent of the initial distribution. In mathematics, a Markov chain, named after Andrey Markov, is a discrete-time stochastic process with the Markov property. ... Irreducible can refer to: irreducible (mathematics) irreducible (philosophy) This is a disambiguation page — a navigational aid which lists other pages that might otherwise share the same title. ... In mathematics, a periodic function is a function that repeats its values, after adding some definite period to the variable. ... In mathematics, a (discrete-time) Markov chain, named after Andrei Markov, is a discrete-time stochastic process with the Markov property. ...

For example, for such a Markov chain Y_k defined on a countable number of states, given the transition matrix P_ij, H(Y) is given by: In mathematics the term countable set is used to describe the size of a set, e. ... In mathematics, a (discrete-time) Markov chain is a discrete-time stochastic process with the Markov property. ...

where μ_i is the stationary distribution of the chain. In mathematics, a (discrete-time) Markov chain, named after Andrei Markov, is a discrete-time stochastic process with the Markov property. ...

A simple consequence of this definition is that the entropy rate of an i.i.d. stochastic process has an entropy rate that is the same as the entropy of any individual member of the process. In probability theory, a sequence or other collection of random variables is independent and identically distributed (i. ... In the mathematics of probability, a stochastic process is a random function. ... Ice melting - classic example of entropy increasing[1] described in 1862 by Rudolf Clausius as an increase in the disgregation of the molecules of the body of ice. ...

References

• Cover, T. and Thomas, J. (1991) Elements of Information Theory, John Wiley and Sons, Inc.

External links

• Systems Analysis, Modelling and Prediction (SAMP), University of Oxford MATLAB code for estimating information-theoretic quantities for stochastic processes.