In information theory, the source coding theorem (Shannon 1948) informally states that: Information theory is the mathematical theory of data communication and storage, generally considered to have been founded in 1948 by Claude E. Shannon. ...

"N i.i.d. random variables each with entropy H(X) can be compressed into more than NH(X) bits with negligible risk of information loss, as N tends to infinity; but conversely, if they are compressed into fewer than NH(X) bits it is virtually certain that information will be lost." (MacKay 2003).

Devising coding strategies to achieve successfully this compression is the basis of the field of entropy encoding. ... For other senses of the term, see entropy (disambiguation). ... BITS may have any of the following meanings: In computer science, bits are binary digits, which may each have the value one or zero. ... An entropy encoding is a coding scheme that assigns codes to symbols so as to match code lengths with the probabilities of the symbols. ...

A more mathematical statement of the theorem is:

Let X be an ensemble with entropy H(X)=H bits. Given ε > 0 and 0 < δ < 1, there exists a positive integer N₀ such that for N > N₀,

where H_δ(X)=log₂|S_δ(X)|; and S_δ(X) is the smallest subset of values of X such that the probability that x is not in S_δ is less than δ. (MacKay 2003).

The source coding theorem is closely related to the asymptotic equipartition property, and the notion of the typical set. The word ensemble can refer to a musical ensemble an ensemble cast (drama) a statistical ensemble in mathematical physics, for example a thermodynamic ensemble a quantum ensemble a fluid mechanical ensemble a Climate Ensemble ensemble forecasting (meteorology) ensemble averaging a distribution ensemble (maths) a neural ensemble a DAB ensemble Ensemble... The asymptotic equipartition property (AEP), or Shannon-McMillan theorem, is a direct consequence of the weak law of large numbers and is used extensively in information theory. ... In information theory, the typical set is a set of sequences whose probability is close to two raised to the negative power of the entropy of their source distribution. ...