This article or section contains information that has not been verified and thus might not be reliable. ...

Introduction

"Gain in entropy always means loss of information, and nothing more" (G. N. Lewis, 1930).

There are close links between the information-theoretic entropy of Shannon and Hartley, usually expressed as H, and the thermodynamic entropy of Clausius and Carnot, usually denoted by S, of a physical system — in particular between the Shannon entropy and the statistical interpretation of thermodynamic entropy, established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s. Lewis in the Berkeley Lab Gilbert Newton Lewis (October 23, 1875-March 23, 1946) was a famous physical chemist. ... 1930 (MCMXXX) is a common year starting on Wednesday. ... Entropy of a Bernoulli trial as a function of success probability. ... Claude Shannon Claude Elwood Shannon (April 30, 1916 – February 24, 2001), an American electrical engineer and mathematician, has been called the father of information theory, and was the founder of practical digital circuit design theory. ... Ralph Vinton Lyon Hartley (November 30, 1888 - May 1, 1970) was an electronics researcher. ... Ice melting - a classic example of entropy increasing Entropy is a concept in thermodynamics, statistical mechanics and information theory. ... Rudolf Clausius - physicist and mathematician Rudolf Julius Emanuel Clausius (January 2, 1822 – August 24, 1888), was a German physicist and mathematician. ... Sadi Carnot Nicolas Léonard Sadi Carnot (June 1, 1796 - August 24, 1832) was a French mathematician and engineer who gave the first successful theoretical account of heat engines, the Carnot cycle, and laid the foundations of the second law of thermodynamics. ... Statistical mechanics is the application of statistics, which includes mathematical tools for dealing with large populations, to the field of mechanics, which is concerned with the motion of particles or objects when subjected to a force. ... Ludwig Boltzmann Ludwig Eduard Boltzmann (Vienna, Austria-Hungary, February 20, 1844 – Duino near Trieste, September 5, 1906) is an Austrian physicist famous for his founding contributions in the fields of statistical mechanics and statistical thermodynamics. ... Josiah Willard Gibbs (February 11, 1839 – April 28, 1903) was an American physical chemist. ... // Events and Trends Technology The invention of the telephone (1876) by Alexander Graham Bell. ...

A quantity defined by the entropy formula for was first introduced by Boltzmann in 1872, in the context of his H-theorem. Boltzmann's definition, based on frequency distribution for a single particle in a gas of like particles, was subsequently reworked by Gibbs into a general formula for the statistical-mechanical entropy (or "mixedupness"), based on the probability distribution p_i for a complete microstate i of the total system: 1872 (MDCCCLXXII) was a leap year starting on Monday (see link for calendar) of the Gregorian calendar or a leap year starting on Wednesday of the 12-day-slower Julian calendar. ... In thermodynamics, the H-theorem describes the increase of entropy of an ideal gas in an irreversible process, solving the Boltzmann equation. ...

The relation between Gibbs's statistical mechanical definition of entropy and Clausius's classical thermodynamical definition is explored further in the article: Thermodynamic entropy. For other uses of the term entropy, see Entropy (disambiguation) The thermodynamic entropy S, often simply called the entropy in the context of thermodynamics, is a measure of the amount of energy in a physical system that cannot be used to do work. ...

It is evident that

where the Shannon entropy H is measured in nats, and the constant of proportionality k_B is Boltzmann's constant. Boltzmann's constant appears here due to the conventional definition of the units of temperature. Beyond that it has no particular fundamental physical significance in the definition of statistical mechanical entropy here. A nat (sometimes also nit or even nepit) is a logarithmic unit of information or entropy, based on natural logarithms and powers of e, rather than the powers of 2 and base 2 logarithms which define the bit. ... The Boltzmann constant (k or kB) is the physical constant relating temperature to energy. ...

In fact, in the view of Jaynes (1957), statistical thermodynamics should be seen as an application of Shannon's information theory: the thermodynamic entropy is interpreted as being an estimate of the amount of further Shannon information needed to define the detailed microscopic state of the system, that remains uncommunicated by a description solely in terms of the macroscopic variables of classical thermodynamics. (See article: MaxEnt thermodynamics). Edwin Thompson Jaynes (July 5th, 1922 – April 30th, 1998) was Wayman Crow Distinguished Professor of Physics at Washington University in St. ... 1957 (MCMLVII) was a common year starting on Tuesday of the Gregorian calendar. ... In physics the MaxEnt school of thermodynamics, initiated with two papers published in the Physical Review by Edwin T. Jaynes in 1957, views statistical mechanics as an inference process: a specific application of inference techniques rooted in information theory, which relate not just to equilibrium thermodynamics, but are general to...

Equilibrium statistical mechanics gives the prescription that the probability distribution which should be assigned for the unknown microstate of a thermodynamic system is that which has maximum Shannon entropy, given that it must also satisfy the macroscopic description of the system. But this is just an application of a quite general rule in information theory, if one wishes to a maximally uninformative distribution. A microstate is a sovereign state having a very small population or very little land area - usually both. ... The principle of maximum entropy is a method for analyzing the available information in order to determine a unique epistemic probability distribution. ...

The thermodynamic entropy, measuring the phase-space spread of this equilibrium distribution, is just this maximum Shannon entropy, multiplied by Boltzmann's constant for historical reasons. The Boltzmann constant (k or kB) is the physical constant relating temperature to energy. ...

A neat physical implication was established by Szilard in 1929, in a refinement of the famous Maxwell's demon thought-experiment. Consider Maxwell's set-up, but with only a single gas particle in a box. If the supernatural demon knows which half of the box the particle is in, it can close a shutter between the two halves of the box, close a piston unopposed into the empty half of the box, and then extract k_BTln2 joules of useful work if the shutter is opened again, and particle isothermally expands back to its original equilibrium occupied volume. In just the right circumstances therefore, the possession of a single bit of Shannon information (a single bit of negentropy in Brillouin's term) really does correspond to a reduction in physical entropy, which theoretically can indeed be parlayed into useful physical work. Leó Szilárd (right) working with Albert Einstein. ... 1929 (MCMXXIX) was a common year starting on Tuesday (link will take you to calendar). ... Maxwells demon is a character in an 1867 thought experiment by the Scottish physicist James Clerk Maxwell, meant to raise questions about the second law of thermodynamics. ... To meet Wikipedias quality standards, this article or section may require cleanup. ...

A corollary is that in storing one bit of previously unstored information in a system, one inevitably potentially reduces the system's entropy by k_Bln2 J K^-1. This is only thermodynamically possible if the storage process releases at least k_BTln2 joules of energy into the system's surroundings. Rolf Landauer (1961) showed that the couterpart of this process can occur as well: an array of ordered bits of memory can become "thermalized," or populated with random data, and in the process cool off its surroundings. N bits would then increase the entropy of the system by Nk_Bln2 as they thermalize. Rolf Landauer (1927 – 1999) was an IBM physicist who in 1961 demonstrated that when information is lost in an irreversible circuit, the information becomes entropy and an associated amount of energy is dissipated as heat. ... 1961 (MCMLXI) was a common year starting on Sunday (the link is to a full 1961 calendar). ...

Heat generation is one of the banes of computer hardware design; so Landauer's principle is interesting as a fundamental physical limit to computation: it is impossible to physically erase one bit of stored information, without the system heating up by an energy of at least k_BTln2 joules. This fundamental limit was one of the original spurs to research into reversible computing, which in turn proved essential for research into quantum computers. Landauers Principle, first argued in 1961 by Rolf Landauer of IBM, holds that any logically irreversible manipulation of information, such as the erasure of a bit or the merging of two computation paths, must be accompanied by a corresponding entropy increase in non-information bearing degrees of freedom of... The term reversible computing refers to any computational process that is (at least to some close approximation) reversible, i. ... Molecule of alanine used in NMR implementation of error correction. ...

The relation between information entropy and thermodynamic entropy has become common currency in physics. Thus Stephen Hawking often speaks of the thermodynamic entropy of black holes in terms of their information content; and it is not surprising that computers must obey the same physical laws that steam engines do, even though they are radically different devices. Stephen Hawking Stephen William Hawking, CH, CBE, FRS, is considered one of the worlds leading theoretical physicists. ... A black hole is a concentration of mass great enough that the force of gravity prevents anything past its event horizon from escaping it except through quantum tunnelling behaviour (known as Hawking Radiation). ...

But it should also be remembered that Gibbs's statistical mechanical entropy is only one application of information theory to physical systems, relevant when the particular 'message' not yet communicated is the underlying microstate of the physical system.

Other physical 'messages' will have their own information entropies. For example, the information rate of a macroscopic physical system obeying stochastic or chaotic behavior can be equal to the information rate of an equivalent Markov process. This entropy is quite likely negligibly tiny and practically quite irrelevant as a contribution to the overall thermodynamic entropy. But if this is the message of interest, then it is the thermodynamic entropy which is irrelevant, and this Shannon information which is everything.

Equivalence of form of defining equations

Discrete case

The defining equation for entropy in the theory of statistical mechanics established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s, is of the form: Ice melting - a classic example of entropy increasing Entropy is a concept in thermodynamics, statistical mechanics and information theory. ... Statistical mechanics is the application of statistics, which includes mathematical tools for dealing with large populations, to the field of mechanics, which is concerned with the motion of particles or objects when subjected to a force. ... Ludwig Boltzmann Ludwig Eduard Boltzmann (Vienna, Austria-Hungary, February 20, 1844 – Duino near Trieste, September 5, 1906) is an Austrian physicist famous for his founding contributions in the fields of statistical mechanics and statistical thermodynamics. ... Josiah Willard Gibbs (February 11, 1839 – April 28, 1903) was an American physical chemist. ... // Events and Trends Technology The invention of the telephone (1876) by Alexander Graham Bell. ...

where p_i is the probability of the microstate i taken from an equilibrium ensemble; which reduces for the special case of the microcanonical ensemble to

where W is the number of microstates, given the fundamental postulate that all the microstates are equiprobable. Statistical mechanics is the application of statistics, which includes mathematical tools for dealing with large populations, to the field of mechanics, which is concerned with the motion of particles or objects when subjected to a force. ...

The defining equation for entropy in the theory of information established by Claude E. Shannon in 1948 is of the form: Entropy of a Bernoulli trial as a function of success probability. ... To meet Wikipedias quality standards, this article or section may require cleanup. ... Claude Elwood Shannon (April 30, 1916 - February 24, 2001) has been called the father of information theory, and was the founder of practical digital circuit design theory. ... 1948 (MCMXLVIII) was a leap year starting on Thursday (the link is to a full 1948 calendar). ...

where p_i is the probability of the message m_i taken from the message space M. This also reduces to

where | M | is the cardinality of the message space M, under the assumption that all the messages are equiprobable. In mathematics, the cardinality of a set is a measure of the number of elements of the set. There are two approaches to cardinality – one which compares sets directly using bijections, injections, and surjections, and another which uses cardinal numbers. ...

In the former case the natural logarithm was taken, and in the latter case the logarithm can also be taken to the natural base, as long as we measure information in nats. In this case we can write The natural logarithm is the logarithm to the base e, where e is equal to 2. ... Logarithms to various bases: is to base e, is to base 10, and is to base 1. ... A nat (sometimes also nit or even nepit) is a logarithmic unit of information or entropy, based on natural logarithms and powers of e, rather than the powers of 2 and base 2 logarithms which define the bit. ...

where k is Boltzmann's constant to express the formal equivalence of these two discrete notions of entropy. The Boltzmann constant (k or kB) is the physical constant relating temperature to energy. ... The word discrete comes from the Latin word discretus which means separate. ...

See Figure 2 of Frank's paper for a striking illustration of this concept, (which he calls "physical information"). Physical information refers generally to the information that is contained in a physical system. ...

This is more than just a formal resemblance of defining equations, however. As Landauer explains, any physical representation of information, such as in data processing equipment, must be somehow embedded in the statistical mechanical degrees of freedom of a physical system. Some of those degrees of freedom are simply taken to represent meaningful information according to the relation just expressed. And as Frank so clearly illustrates, it is rather arbitrary which of those degrees of freedom we take to represent "known information." For example, the conversion of general thermodynamic entropy (if it is indeed possible) to "known information" would be a very good source of random numbers, which are quite useful for many computational purposes.

Continuous case

Boltzmann's H-function likewise formally resembles Shannon's entropy in the continuous case. Basically, the H-function can be expressed, up to sign convention, as the information-theoretic joint entropy of the continuous probability distributions of the coordinates and momenta of the particles under consideration. In thermodynamics, the H-theorem describes the increase of entropy of an ideal gas in an irreversible process, solving the Boltzmann equation. ... Look up Up to on Wiktionary, the free dictionary In mathematics, the phrase up to xxxx indicates that members of an equivalence class are to be regarded as a single entity for some purpose. ... The joint entropy is an entropy measure used in information theory. ... In mathematics and statistics, a probability distribution, more properly called a probability density, assigns to every interval of the real numbers a probability, so that the probability axioms are satisfied. ...

But this connection remains murky, and is not nearly so clear and undeniable as the discrete case. It is complicated by the fact that the joint entropy in the continuous case fails to be invariant under linear transformation (an important property of the mutual information). This makes the definition of the H-function dependent on a choice of the units of measure used for the coordinates and momenta. Invariant may have meanings invariant (computer science), such as a combination of variables not altered in a loop invariant (mathematics), something unaltered by a transformation invariant (music) invariant (physics) conserved by system symmetry This is a disambiguation page — a navigational aid which lists other pages that might otherwise share the... In mathematics, a linear transformation (also called linear map or linear operator) is a function between two vector spaces that preserves the operations of vector addition and scalar multiplication. ... In probability theory and, in particular, information theory, the mutual information, or transinformation, of two random variables is a quantity that measures the mutual dependence of the two variables. ...

Hirschman showed in 1957, however, that Heisenberg's uncertainty principle can be expressed as a particular lower bound on the sum of the entropies of the observable probability distributions of a particle's position and momentum, when they are expressed in Planck units. (One could speak of the "joint entropy" of these distributions by considering them independent, but since they are not jointly observable, they cannot be considered as a joint distribution.) In quantum physics, the Heisenberg uncertainty principle, sometimes called the Heisenberg indeterminacy principle, expresses a limitation on accuracy of (nearly) simultaneous measurement of observables such as the position and the momentum of a particle. ... In physics, Planck units are a system of physical units of measurement. ...

The von Neumann-Landauer bound

A theoretical application of this formal equivalence of thermodynamic entropy and information-theoretic entropy in the discrete case yields a lower bound on the amount of heat generated by an irreversible computational process, known as the von Neumann-Landauer bound.

Rolf Landauer argued in a 1961 paper that computational operations that are logically irreversible are also physically irreversible in the sense that reversing them would break the second law of thermodynamics. This result is known as Landauer's principle. In that paper, he also quantified the minimum net increase in thermodynamic entropy that must take place for an operation in which one bit of information is lost: kln2. This increase in entropy that occurs for an irreversible bit operation must be expelled as kTln2 heat to the environment at an absolute temperature T. The factor ln2 comes from the fact that 1 bit = (ln 2) nat. Rolf Landauer (1927 – 1999) was an IBM physicist who in 1961 demonstrated that when information is lost in an irreversible circuit, the information becomes entropy and an associated amount of energy is dissipated as heat. ... The second law of thermodynamics states that The Second Law is a statistical law and thus applicable only to macroscopic systems. ... Landauers Principle, first argued in 1961 by Rolf Landauer of IBM, holds that any logically irreversible manipulation of information, such as the erasure of a bit or the merging of two computation paths, must be accompanied by a corresponding entropy increase in non-information bearing degrees of freedom of... This article is about the unit of information. ... A nat (sometimes also nit or even nepit) is a logarithmic unit of information or entropy, based on natural logarithms and powers of e, rather than the powers of 2 and base 2 logarithms which define the bit. ...

This principle is important because it establishes physical limits to computation. There are ideas to implement schemes of reversible computing, but interestingly, Landauer argued in his paper that such schemes would be impractical because of a large increase in the amount of memory that would be required, and moreover the heat that would be generated by the irreversible step of initializing this memory would offset any heat savings realized by the implementation of reversibility. Nevertheless, much research has been devoted to the theory of reversible computing. Bennett in particular established some results that indicate that the time and space overhead for a reversible computer may not be too great. The term reversible computing refers to any computational process that is (at least to some close approximation) reversible, i. ... The term reversible computing refers to any computational process that is (at least to some close approximation) reversible, i. ... Charles H. Bennett Charles H. Bennett is an IBM Fellow at IBM Research. ...

This has been applied to the paradox of Maxwell's demon which would need to process information to reverse thermodynamic entropy; but erasing that information, to begin again, exactly balances out the thermodynamic gain that the demon would otherwise achieve. Maxwells demon is a character in an 1867 thought experiment by the Scottish physicist James Clerk Maxwell, meant to raise questions about the second law of thermodynamics. ...

Black holes

Stephen Hawking often speaks of the thermodynamic entropy of black holes in terms of their information content. Do black holes destroy information? Didn't Hawking lose a bet about that one? See Black hole entropy and Black hole information paradox. Stephen Hawking Stephen William Hawking, CH, CBE, FRS, is considered one of the worlds leading theoretical physicists. ... A black hole is a concentration of mass great enough that the force of gravity prevents anything past its event horizon from escaping it except through quantum tunnelling behaviour (known as Hawking Radiation). ... Black hole entropy is entropy carried by a black hole. ... The black hole information paradox results from the combination of quantum mechanics and general relativity. ...

The Fluctuation Theorem

The fluctuation theorem provides a mathematical justification of the second law of thermodynamics under these principles, and precisely defines the limitations of the applicability of that law to the microscopic realm of individual particle movements. The second law of thermodynamics stands in apparent contradiction with the time reversible equations of motion for classical and quantum systems. ... The second law of thermodynamics states that The Second Law is a statistical law and thus applicable only to macroscopic systems. ...

Topics of recent research

Is information quantized?

In 1995, Tim Palmer signalled two unwritten assumptions about Shannon's definition of information that may make it inapplicable as such to quantum mechanics: 1995 (MCMXCV) was a common year starting on Sunday of the Gregorian calendar. ... For a non-technical introduction to the topic, please see Introduction to Quantum mechanics. ...

• The supposition that there is such a thing as an observable state (for instance the upper face a die or a coin) before the observation begins

• The fact that knowing this state does not depend on the order in which observations are made (commutativity)

The article Conceptual inadequacy of the Shannon information in quantum measurement [1], published in 2001 by Anton Zeilinger [2] and Caslav Brukner, synthesized and developed these remarks. The so-called Zeilinger's principle suggests that the quantization observed in QM could be bound to information quantization (one cannot observe less than one bit, and what is not observed is by definition "random"). Anton Zeilinger Anton Zeilinger (born on 20 May 1945 in Ried im Innkreis, Austria) is a professor of physics at the University of Vienna, previously Innsbruck. ...

But these claims remain highly controversial. For a detailed discussion of the applicability of the Shannon information in quantum mechanics and an argument that Zeilinger's principle cannot explain quantization, see Timpson [3] 2003 [4] and also Hall 2000 [5] and Mana 2004 [6]

For a tutorial on quantum information see [7].

See also

• Thermodynamic entropy
• Information entropy
• Thermodynamics
• Statistical mechanics
• Information theory
• Physical information
• Fluctuation theorem
• Black hole entropy
• Black hole information paradox

Ice melting - a classic example of entropy increasing Entropy is a concept in thermodynamics, statistical mechanics and information theory. ... Entropy of a Bernoulli trial as a function of success probability. ... Thermodynamics (from the Greek thermos meaning heat and dynamis meaning power) is a branch of physics that studies the effects of changes in temperature, pressure, and volume on physical systems at the macroscopic scale by analyzing the collective motion of their particles using statistics. ... Statistical mechanics is the application of statistics, which includes mathematical tools for dealing with large populations, to the field of mechanics, which is concerned with the motion of particles or objects when subjected to a force. ... To meet Wikipedias quality standards, this article or section may require cleanup. ... Physical information refers generally to the information that is contained in a physical system. ... The second law of thermodynamics stands in apparent contradiction with the time reversible equations of motion for classical and quantum systems. ... Black hole entropy is entropy carried by a black hole. ... The black hole information paradox results from the combination of quantum mechanics and general relativity. ...

External Links

• Entropy is Simple...If You Avoid the Briar Patches. Very dismissive of any link between information-theoretic and thermodynamic entropy.

References

• C. H. Bennett, "Logical reversibility of computation," IBM Journal of Research and Development, vol. 17, no. 6, pp. 525-532, 1973.
• Leon Brillouin, Science and Information Theory, Mineola, N.Y.: Dover, [1956, 1962] 2004. ISBN 0486439186
• Michael P. Frank, "Physical Limits of Computing", Computing in Science and Engineering, 4(3):16-25, May/June 2002.
• Andreas Greven, Gerhard Keller, and Gerald Warnecke, editors. Entropy, Princeton University Press, 2003. ISBN 0691113386. (A highly technical collection of writings giving an overview of the concept of entropy as it appears in various disciplines.)
• I. Hirschman, A Note on Entropy, American Journal of Mathematics, 1957.
• R. Landauer, Information is Physical Proc. Workshop on Physics and Computation PhysComp'92 (IEEE Comp. Sci.Press, Los Alamitos, 1993) pp. 1-4.
• R. Landauer, Irreversibility and Heat Generation in the Computing Process IBM J. Res. Develop. Vol. 5, No. 3, 1961
• H. S. Leff and A. F. Rex, Editors, Maxwell's Demon: Entropy, Information, Computing, Princeton University Press, Princeton, NJ (1990). ISBN 069108727X
• Claude E. Shannon. A Mathematical Theory of Communication. Bell System Technical Journal, July/October 1948.