Re: Entropy and information

> On 2/27/2012 10:59 AM, Evgenii Rudnyi wrote:
>> On 27.02.2012 00:13 meekerdb said the following:
>>> On 2/26/2012 5:58 AM, Evgenii Rudnyi wrote:
>>>> I have written a summary for the discussion in the subject:
>>>>
>>>> http://blog.rudnyi.ru/2012/02/entropy-and-information.html
>>>>
>>>> No doubt, this is my personal viewpoint. If you see that I have
>>>> missed something, please let me know.
>>>
>>> I think you are ignoring the conceptual unification provided by
>>> information theory and statistical mechanics. JANAF tables only
>>> consider the thermodynamic entropy, which is a special case in which
>>> the macroscopic variables are temperature and pressure. You can't
>>> look up the entropy of magnetization in the JANAF tables.
>>
>> I do not get your point. JANAF Tables have been created to solve a
>> particular problem. If you need change in concentration, surface
>> effects, magnetization effects, you have to extend the JANAF Tables.
>> And this has been to solve particular problems. Experimental
>> thermodynamics is not limited to JANAF Tables. For example, the
>> databases in Thermocalc already include dependence on concentration.
>
> And you don't get my point. Of course all forms of entropy can be
> measured and tabulated, but the information theory viewpoint shows how
> they are unified by the same concept.
>
>>
>>> Yet
>>> magnetization of small domains is how information is stored on hard
>>> disks, c.f. Donald McKay's book "Information Theory, Inference, and
>>> Learning Algorithm" chapter 31.
>>
>> Do you mean that when we consider magnetization, then the entropy
>> become subjective, context-dependent, and it will be finally filled
>> with information?
>
> It is context dependent in that we consider the magnetization. What does
> the JANAF table assume about the magnetization of the materials it
> tabulates?
>
>>
>>> Did you actually read E. T. Jaynes 1957 paper in which he introduced
>>> the idea of basing entropy in statistical mechanics (which you also
>>> seem to dislike) on information? He wrote "The mere fact that the
>>> same mathematical expression -SUM[p_i log(p_i)] occurs in both
>>> statistical mechanics and in information theory does not in itself
>>> establish a connection between these fields. This can be done only by
>>> finding new viewpoints from which the thermodynamic entropy and
>>> information-theory entropy appear as the same /concept/." Then he
>>
>> I have missed this quote, I have to add it. In general, the first
>> Jaynes's paper is in a way reasonable. I wanted to better understand
>> it, as I like maximum likelihood, I have been using it in my own
>> research a lot. However, when I have read in Jaynes's second paper the
>> following (two quotes below), I gave up.
>>
>> “With such an interpretation the expression “irreversible process”
>> represents a semantic confusion; it is not the physical process that
>> is irreversible, but rather our ability to follow it. The second law
>> of thermodynamics then becomes merely the statement that although our
>> information as to the state of a system may be lost in a variety of
>> ways, the only way in which it can be gained is by carrying out
>> further measurements.”
>>
>> “It is important to realize that the tendency of entropy to increase is
>> not a consequence of the laws of physics as such, … . An entropy
>> increase may occur unavoidably, due to our incomplete knowledge of the
>> forces acting on a system, or it may be entirely voluntary act on our
>> part.”
>>
>> This I do not understand. Do you agree with these two quotes? If yes,
>> could you please explain, what he means?
>
> Yes. The physical processes are not irreversible. The fundamental
> physical laws are time reversible. The free-expansion of a gas is
> *statistically* irreversible because we cannot follow the individual
> molecules and their correlations, so when we consider only the
> macroscopic variables of pressure, density, temperature,... it seems
> irreversible. In very simple systems we might be able to actually follow
> the microscopic evolution of the state, but we can choose to ignore it
> and calculate the entropy increase as though this information were lost.
> Whether and how the information is lost is the crux of the measurement
> problem in QM. Almost everyone on this list assumes Everett's multiple
> worlds interpretation in which the information is not lost but is
> divided up among different continuations of the observer.
>
>>
>>> goes on to show how the principle of maximum entropy can be used to
>>> derive statistical mechanics. That it *can* be done in some other
>>> way, and was historically as you assert, is not to the point. As an
>>> example of how the information view of statistical mechanics extends
>>> its application he calculates how much the spins of protons in water
>>> would be polarized by rotating the water at 36,000rpm. It seems you
>>> are merely objecting to "new viewpoints" on the grounds that you can
>>> see all that you /want/ to see from the old viewpoint.
>>
>>> Your quotation of Arnheim, from his book on the theory of entropy in
>>> art, just shows his confusion. The Shannon information, which is
>>> greatest when the system is most disordered in some sense, does not
>>> imply that the most disordered message contains the greatest
>>> information. The Shannon information is that information we receive
>>> when the *potential messages* are most disordered. It's a property of
>>> an ensemble or a channel, not of a particular message.
>>
>> It is not a confusion of Arnheim. His book is quite good.
>
> Then why doesn't he know that Shannon's information does not refer to
> particular messages?
>
>> To this end, let me quote your second sentence in your message.
>>
>> > I think you are ignoring the conceptual unification provided by
>> > information theory and statistical mechanics.
>>
>> You see, I would love to understand the conceptual unification.
>
> I find that doubt this. If you really loved to understand it there is
> lots of material online as well as good books which Russell and I have
> suggested.
>
>> To this end, I have created many simple problems to understand this
>> better. Unfortunately you do not want to discuss them, you just saying
>> general words but you do not want to apply it to my simple practical
>> problems. Hence it is hard for me to understand you.
>>
>> If to speak about confusion, just one example. You tell that the
>> higher temperature the more information the system has. Yet, engineers
>> seems to be unwilling to employ this knowledge in practice. Why is
>> that? Why engineers seem not to be impressed by the conceptual
>> unification?
>
> I'm not an expert on this subject and it has been forty years since I
> studied statistical mechanics, which is why I prefer to refer you to
> experts. Engineers are generally not impressed with conceptual
> unification; they are interested in what can be most easily and reliably
> applied. RF engineers generally don't care that EM waves are really
> photons. Structural engineers don't care about interatomic forces, they
> just look yield strength in tables. Engineers are not at all concerned
> with 'in principle' processes which can only be realized in carefully
> contrived laboratory experiments. But finding unifying principles is the
> job of physicists.
>
> Brent
>