[] relevant information

[Mikhail.Prokopenko]



Hi,

 

In trying to discuss IDSO, I assume information as the reduction of uncertainty (as pointed by Stan already) - following traditional information-theoretic formalisms (Shannon, Renyi, etc.). There was no intention to take the general "information systems" view: as Hussein pointed out, "everything"-driven self-organisation would be very hard to understand :)

 

1) It would be useful to have a look at the following quote from Adami (in the context of genetic information):

C. Adami. What is complexity? Bioessays, 24(12):1085–1094, 2002

http://faculty.kgi.edu/adami/BE2002.pdf

 

================

Randomness is in some ways the ``flip side’’ of information, and is called entropy in information theory.(15) Entropy is a measure of potential knowledge, or if applied to a sequence, a measure of how much information a sequence could hold, and thus quantifies our uncertainty about the genetic identity of a randomly selected individual from a pool. It is useful to think of sequence entropy as the length of a tape, while information is the length of tape containing recordings. Measurement (i.e., recording) turns empty tape into filled tape; entropy into information. As we shall see, this is what happens during adaptation, and it is the force that drives the increase of complexity.

Information is a statistical form of correlation, and thus requires, mathematically and intuitively, a reference to the system that the information is about. The sequence on your information-filled tape allows you to make predictions about the state of the system that the sequence is information about. This predictive capability implies that your sequence and the system have something in common, that they are correlated. Your sequence will most likely not make predictions about any other system (unless the systems are very similar). If you do not know which system your sequence refers to, then whatever
is on it cannot be considered information. Instead, it is potential information (a.k.a. entropy). This is the fundamental difference between entropy and information, often misrepresented in the literature.(16)

 

15. Shannon CE, Weaver W. The Mathematical Theory of Communication.
Urbana: University of Illinois Press. 1949.

16. Adami C. Information theory in molecular biology. 2002

================

 

What I like about Adami's work is the clear separation between entropy and information, and the elegant way to contextualise the relevance via "physical complexity". The latter is defined, for a population X (an ensemble of sequences), in relation to a specific environment Z, as mutual information:

 I(X,Z) = Hmax − H(X|Z),

where Hmax is the entropy in the absence of selection, i.e. the unconditional entropy of a population of sequences, and H(X|Z) is the conditional entropy of X given Z, i.e. the diversity tolerated by selection in the given environment.

 

2) There is also work by Daniel Polani et al. that dates back to at least 2001 (Daniel, please correct me if I'm wrong):

    D. Polani, J. T. Kim, and T. Martinetz: An Information-Theoretic Approach for the Quantification of Relevance. In: J. Kelemen and P. Sosik (eds.), Advances in Artificial Life (Proc. 6th European Conference on Artificial Life, Prague, September 10-14), LNCS. Springer 2001

http://homepages.feis.herts.ac.uk/~comqdp1/publications/files/RI2.pdf

and

    Polani, D., Nehaniv, C., Martinetz, T., and Kim, J. T., (2006). Relevant Information in Optimized Persistence vs. Progeny Strategies. In M.Rocha, L., Bedau, M., Floreano, D., Goldstone, R., Vespignani, A., and Yaeger, L., editors, (2006). Proc. Artificial Life X.

http://homepages.feis.herts.ac.uk/~comqdp1/publications/files/polani_alife_2006.pdf

 

building up on Tishby et al. work (1999) on relevant information and the information bottleneck method:

    Tishby, N., Pereira, F. C., and Bialek, W. (1999). The information bottleneck method. In Proceedings of 37th Annual Allerton Conference on Communication, Control and Computing, Illinois.

 

Thanks,

Mikhail

 

 

[Stanley N. Salthe]

Folks, for the record, regarding the entropy formulations below (Hmax,
etc.) we might note the existence of the book 'Evolution as Entropy' by Dan
Brooks and Ed Wiley (l988, 2nd Ed) Univ Chivago Press.  Following works in
cosmology, they discuss information changes in GROWNIG or expanding
systems, focusingon organic evolution.

STAN

>Ôªø    Hi,   In trying to discuss IDSO, I assume  information as the
>reduction of uncertainty (as pointed by Stan already) -  following
>traditional information-theoretic formalisms (Shannon, Renyi, etc.).
>There was no intention to take the general "information systems" view: as
>Hussein pointed out, "everything"-driven  self-organisation would be very
>hard to understand  :)   1) It would  be useful to have a look at the
>following quote from Adami (in the context of  genetic information): C.
>Adami.  What is complexity? Bioessays, 24(12):1085’Äì1094, 2002
>http://faculty.kgi.edu/adami/BE2002.pdf   ================ Randomness  is
>in some ways the ``flip side’Äô’Äô of information, and is called entropy
>in  information theory.(15) Entropy is a measure of potential knowledge,
>or if  applied to a sequence, a measure of how much information a sequence
>could hold,  and thus quantifies our uncertainty about the genetic
>identity of a randomly  selected individual from a pool. It is useful to
>think of sequence entropy as  the length of a tape, while information is
>the length of tape containing  recordings. Measurement (i.e., recording)
>turns empty tape into filled tape;  entropy into information. As we shall
>see, this is what happens during  adaptation, and it is the force that
>drives the increase of  complexity.
>Information is a statistical form of correlation,  and thus requires,
>mathematically and intuitively, a reference to the system  that the
>information is about. The sequence on your information-filled tape  allows
>you to make predictions about the state of the system that the sequence
>is information about. This predictive capability implies that your
>sequence and  the system have something in common, that they are
>correlated. Your sequence  will most likely not make predictions about any
>other system (unless the systems  are very similar). If you do not know
>which system your sequence refers to, then  whatever
>is on it cannot be considered information. Instead, it is potential
>information (a.k.a. entropy). This is the fundamental difference between
>entropy  and information, often misrepresented in the literature.(16)  
>15. Shannon  CE, Weaver W. The Mathematical Theory of Communication.
>Urbana: University of  Illinois Press. 1949.
>16. Adami C. Information theory in molecular biology. 2002
>================   What I like about Adami's work is the clear  separation
>between entropy and information, and the elegant way to contextualise  the
>relevance via "physical complexity".  The latter is defined, for a
>population X (an ensemble of sequences), in  relation to a specific
>environment Z, as mutual information:
>  I(X,Z) = Hmax ’àí  H(X|Z),
> where Hmax is the entropy in the absence of  selection, i.e. the
>unconditional entropy of a population of sequences, and  H(X|Z) is the
>conditional entropy of X given Z, i.e. the diversity tolerated by
>selection in the given environment.   2) There is also work by Daniel
>Polani et  al. that dates back to at least 2001 (Daniel, please correct me
>if I'm  wrong):     D. Polani, J. T. Kim, and  T. Martinetz: An
>Information-Theoretic Approach for the Quantification of  Relevance. In:
>J. Kelemen and P. Sosik (eds.), Advances in Artificial Life  (Proc. 6th
>European Conference on Artificial Life, Prague, September 10-14),  LNCS.
>Springer 2001
>http://homepages.feis.herts.ac.uk/~comqdp1/publications/files/RI2.pdf and
>    Polani, D., Nehaniv, C.,  Martinetz, T., and Kim, J. T., (2006).
>Relevant Information in Optimized  Persistence vs. Progeny Strategies. In
>M.Rocha, L., Bedau, M., Floreano, D.,  Goldstone, R., Vespignani, A., and
>Yaeger, L., editors, (2006). Proc. Artificial  Life X.
>http://homepages.feis.herts.ac.uk/~comqdp1/publications/files/polani_alife_2006.
>pdf   building up on Tishby et al. work (1999) on  relevant information
>and the information bottleneck method:     Tishby, N., Pereira, F.  C.,
>and Bialek, W. (1999). The information bottleneck method. In Proceedings
>of 37th Annual Allerton Conference on Communication, Control and
>Computing,  Illinois.   Thanks, Mikhail