The Definitions of Entropy |

- Introduction: Entropy Defined
- Entropy & Classical Thermodynamics
- Entropy & Physical Chemistry
- Entropy & Statistical Mechanics
- Entropy & Quantum Mechanics
- Entropy & Information Theory
- Is Entropy a Measure of "Disorder"?

The popular literature is littered
with articles, papers, books, and various & sundry other sources, filled to overflowing with
prosaic explanations of entropy. But it should be remembered that entropy, an idea born from
classical thermodynamics, is a quantitative entity, and not a qualitative one. That means that
entropy is not something that is fundamentally intuitive, but something that is fundamentally
defined via an equation, via mathematics applied to physics. Remember in your various travails,
that *entropy is what the equations define it to be*. There is no such thing as an "entropy",
without an equation that defines it.

Entropy was born as a state variable in classical thermodynamics. But the advent of statistical
mechanics in the late 1800's created a new look for entropy. It did not take long for Claude
Shannon to borrow the Boltzmann-Gibbs formulation of entropy, for use in his own work, inventing
much of what we now call *information theory*. My goal here is to shwo how entropy works,
in all of these cases, not as some fuzzy, ill-defined concept, but rather as a clearly defined,
mathematical & physical quantity, with well understood applications.

**Entropy and Classical Thermodynamics**

Classical thermodynamics developed during the 19th century, its primary architects being
Sadi Carnot,
Rudolph Clausius,
Benoit Claperyon,
James Clerk Maxwell, and
William Thomson (Lord Kelvin).
But it was Clausius who first explicitly advanced the idea of *entropy*
(*On Different Forms of the Fundamental Equations of the Mechanical Theory of Heat*, 1865;
*The Mechanical Theory of Heat*, 1867). The concept was expanded upon by Maxwell
(*Theory of Heat*, Longmans, Green & Co. 1888; Dover reprint, 2001). The specific definition, which
comes from Clausius, is as shown in equation 1 below.

Equation 1

In equation 1, **S** is the *entropy*, **Q** is the *heat content*
of the system, and **T** is the *temperature* of the system. At this time,
the idea of a gas being made up of tiny molecules, and temperature representing
their average kinetic energy, had not yet appeared. Carnot & Clausius thought of
heat as a kind of fluid, a conserved quantity that moved from one system to the
other. It was Thomson who seems to have been the first to explicity recognize
that this could not be the case, because it was inconsistent with the manner
in which mechanical work could be converted into heat. Later in the 19th century,
the molecular theory became predominant, mostly due to Maxwell, Thomson and
Ludwig Boltzmann, but we will cover that story later. Suffice for now to point
out that what they called *heat content*, we would now more commonly call
the *internal heat energy*.

The temperature of the system is an explicit part of this classical definition
of entropy, and a system can only have "a" temperature (as opposed to several
simultaneous temperatures) if it is in thermodynamic equilibrium. So, *entropy
in classical thermodynamics is defined only for systems which are in thermodynamic
equilibrium*.

As long as the temperature is therefore a constant, it's a simple enough exercise to differentiate equation 1, and arrive at equation 2.

Equation 2

Here the symbol "" is a representation of a finite increment, so
that **S** indicates a "change" or "increment" in
**S**, as in **S = S _{1} - S_{2}**, where

Clausius and the others, especially Carnot, were much interested in the ability to
convert mechanical work into heat energy, and vice versa. This idea can lead us to
an alternate form for equation 2, that will be useful later on. Suppose you pump
energy, **U**, into a system, what happens? Part of the energy
goes into the internal heat content, **Q**, making **Q** a
positive quantity, but not all of it. Some of that energy could easily be expressed
as an amount of mechanical work done by the
system (**W**, such as a hot gas pushing against a piston in
a car engine). So that
**Q = U - W**, where
**U** is the energy input to the system, and **W**
is the part of that energy that goes into doing work. The difference between them is the amount
of energy that does not participate in the work, and goes into the heat resevoir as
**Q**. So a simple substitution allows equation 2 to be re-written
as equation 3.

Equation 3

This alternate form of the equation works for heat taken out of a system (**U**
is negative) or work done on a system (**W** is negative), just as well. So
now we have a better idea of the classical relation between work, energy and entropy. Before we
go on to the more advanced topic of statistical mechanics, we will take a useful moment to apply
this to classical chemistry.

**Entropy and Physical Chemistry**

At the same time that engineers & physicists were laying the foundations for thermodynamics, the chemists were not being left out. Classical entropy plays a role in chemical reactions, and that role is exemplified in equation 4 below.

Equation 4

Of course, this looks just like equation 3 with different letters, and so it is. Here, we are
not much interested in the physicists approach of describing the state of a "static" system,
as does equation 1. The real interest for the chemist, is to predict whether or not a given
chemical reaction will go. In equation 4, **H** is the *enthalpy*, and **F** is
the *free energy* (also known as the *Gibb's free energy*). Likewise, **H**
and **F** are incremental variations of those quantities, and **S**
is an incremental change in the entropy of the chemical system, in the event of a chemical reaction.

A little algebra, leading to equation 5, will maybe make things just a little easier to see.

Equation 5

The significance of this equation is that it is the value of **F** which
tells you whether any give chemical reaction will go forward spontaneously, or whether it needs
to be pumped. The enthalpy, **H**, is the heat content of the system, and so the change in
enthalpy, **H**, is the change in heat content of the system. If that
value is smaller than **TS**, then **F** will
be negative, and the reaction will proceed spontaneously; the **TS**
term represents the ability to do the work required to make the reaction happen. However,
if **F** is positive, such that **H** is
greater than **TS**, then the reaction will not happen spontaneously;
we still need at least **F** worth of energy to make it happen.

Note that a positive free energy does not mean that the reaction will not happen, only that
it will not happen *spontaneously* in the given environment. It can still be pushed
or pumped into happening by adding energy, or setting the reaction in a higher temperature
environment, making **T** larger as well as **TS**, and perhaps
driving it far enough to make **F** negative.

**Entropy and Statistical Mechanics**

In the later 1800's,
Maxwell,
Ludwig Boltzmann and
Josiah Willard Gibbs
extended the ideas of classical thermodynamics, through the new "molecular theory" of gases, into
the domain we now call *statistical mechanics*. In classical thermodynamics, we deal with single
extensive systems, whereas in statistical mechanics we recognize the role of the tiny constituents of
the system. The temperature, for instance, of a system defines a *macrostate*, whereas the kinetic
energy of each molecule in the system defines a *microstate*. The macrostate variable, temperature,
is recognized as an expression of the average of the microstate variables, an average kinetic energy
for the system. Hence, if the molecules of a gas move faster, they have more kinetic energy, and the
temperature naturally goes up.

Equation 6 below is the general form of the definition of entropy in statistical mechanics, as
first derived by Boltzmann. You can see Boltzmann's own derivation in his *Lectures
on Gas Theory* (available as a Dover reprint), but more modern treatments might be easier to
follow, such as *Statistical Physics* by Gregory H. Wannier, or *The Principles of Statistical
Mechanics* by Richard C. Tolman (both also available as Dover reprints).

Equation 6

In this equation, **P _{i}** is the probability that particle "

Contrary to the definition seen in equation 1, neither the temperature nor the heat energy appear explicitly in this equation. However, the restriction that all of the microstate probabilities must be calculated for the same macrostate, assures that, as in the earlier case, the system must be in a state of thermal equlibrium.

Equation 6 treats the microstate probabilities individually. However, if all of the probabilities are the same, then we can simplify equation 6 to equation 7.

Equation 7

In this simplified form, the only thing we have to worry about is "**N**", which is
the total number of microstates available to the system. Be careful to note that this is
*not* the total number of particles, but rather the total number of microstates that
the particles could occupy, with the constraint that all such microstate collections
would show the same macrostate.

I have separated quantum mechanics from statistical mechanics, to avoid confusion,
and to avoid the implication that something important may have been overlooked.
However, since the two both deal intimately with statistics & probabilities, it
should come as no surprise that entropy is handled by the two disciplines in much
the same way. The quantum mechanical definition of entropy is identical to that
given for statistical mechanics, in equation 6. The only real
difference is in how the probabilities are calculated. Quantum mechanics has its
own, peculiar rules for doing that, but they are not relevant to the fundamental
definition of entropy. As in the previous cases, the **P _{i}** are
microstate probabilites, and they must all be calculated for the same macrostate.

**Entropy and Information Theory**

The work done, primarily by
Boltzmann &
Gibbs,
on the foundations of statistical mechanics, is
of profound significance that can hardly be overestimated. In the hands of
Clausius
and his contemporarys, entropy was an important, but strictly thermodynamic property. Outside of
physics, it simply had no meaning. But the mathematical foundations of statistical mechanics
are applicable to any statistical system, regardless of its status as a thermodynamic
system. So it is by the road of statistical mechanics, that we are able to talk about
entropy in fields outside of thermodynamics, and even outside of physics *per se*.

Perhaps the first major excursion of entropy into new domains, comes at the hands of Claude Shannon, widely recognized as the father of modern communication & information theory (his classical 1948 paper A Mathematical Theory of Communication is on the web).

Equation 8

If this looks familiar, it's not an accident. It's quite the same as equation 6 above,
the definition of entropy in statistical mechanics. In *A Mathematical Theory of Communication*,
appendix 2, Shannon proves his Theorem 2, that this Boltzmann entropy is the only function
which satisfy's the requirements for a function to measure the uncertainty in a message
(where a "message" is a string of binary bits). In this case, the constant **k** is
recognized as only setting the units; it is arbitrary, and can be set equal to exactly 1
without any loss of generality (see the discussion in Shannon's paper, begining with section
6 "Choice, uncertainty and entropy"). In this case the probabilty **P _{i}** is
the probability for the value of a given bit (usually a binary bit, but not necessarily).

In Shannon information theory, the entropy is a measure of the uncertainty over the true content of a message, but the task is complicated by the fact that successive bits in a string are not random, and therefore not mutually independent, in a real message. Also note that "information" is not a subjective quantity here, but rather an objective quantity, measured in bits.

So far, we have looked at entropy in its most common, and well known forms. Most ordinary applications will use one of these entropies. But, there are other forms of entropy beyond what I have shown. For instance, Brazilian Mathematician Constantino Tsallis has derived a generalized form for entropy, which reduces to the Boltzmann-Gibbs entropy in our equation 6, as a special case, but can also be used to describe the entropy of a system for which our equation 6 would not work (see "Justifying the Tsallis Formalism"). Hungarian mathematician Alfréd Rényi was able to construct the proper entropy for fractal geometries (see "The world according to Rényi: thermodynamics of fractal systems""). There are others besides these, but the entropies of Tsallis & Rényi are the ones that seem to be under the most active current consideration.

These generalized forms of entropy, of relatively recent origin, serve to show that "entropy" is not just an old friend that we know quite well, as in classical thermodynamics, but also a concept that is rich in new ideas & scientific directions.

**Is Entropy a Measure of "Disorder"?**

Let us dispense with at least one popular myth: "*Entropy is disorder*" is
a common enough assertion, but commonality does not make it right.
Entropy *is not* "disorder", although the two can be related to one another.
For a good lesson on the traps and pitfalls of trying to assert what entropy is, see *Insight into entropy* by
Daniel F. Styer, American Journal of Physics
68(12): 1090-1096 (December 2000). Styer uses liquid crystals to illustrate examples of increased entropy
accompanying increased "order", quite impossible in the *entropy is disorder* worldview.
And also keep in mind that "order" is a subjective term, and as such it is subject to the whims of
interpretation. This too mitigates against the idea that entropy and "disorder" are always the same, a fact well
illustrated by Canadian physicist Doug Craigen, in his online essay
"Entropy, God and Evolution".

The easiest answer to the question, "*What is entropy?*", is to reiterate something I said in
the introduction: **Entropy is what the equations define it to be**. You can interpret those
equations to come up with a prosey explanation, but remember that the prose & the equations have
to match up, because the equations give a firm, mathematical definition for entropy, that just
won't go away. In classical thermodynamics, the entropy of a system is the ratio of heat content
to temperature (equation 1), and the change in entropy represents the amount
of energy input to the system which does not participate in mechanical work done by the system
(equation 3). In statistical mechanics, the interpretation is more general perhaps,
where the entropy becomes a function of statistical probability. In that case the entropy is a
measure of the probability for a givem macrostate, so that a high entropy indicates a high
probability state, and a low entropy indicates a low probability state (equation 6).

Entropy is also sometimes confused with *complexity*, the idea being that a more complex
system must have a higher entropy. In fact, that is in all liklihood the opposite of reality.
A system in a highly complex state is probably far from equilibrium and in a low entropy
(improbable) state, where the equilibrium state would be simpler, less complex, and higher
entropy.

Move on to the second law of thermodynamics

Go back to The Collected Writings of Tim Thompson

Go all the way to Tim Thompson's Home Page

**Page dated 19 February 2002**