Article
Full entry |
PDF
(1.1 MB)
Feedback
PDF
(1.1 MB)
Feedback
Keywords:
multi-information; exponential family; relative entropy; pair- interaction; infomax principle; Boltzmann machine; neural networks
multi-information; exponential family; relative entropy; pair- interaction; infomax principle; Boltzmann machine; neural networks
Summary:
Stochastic interdependence of a probability distribution on a product space is measured by its Kullback–Leibler distance from the exponential family of product distributions (called multi-information). Here we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure. Based on a detailed description of the structure of probability distributions with globally maximal multi-information we obtain our main result: The exponential family of pure pair-interactions contains all global maximizers of the multi-information in its closure.
References:
[1] Aarts E., Korst J.: Simulated Annealing and Boltzmann Machines. Wiley, New York 1989 MR 0983115 | Zbl 0674.90059
[2] Ackley D. H., Hinton G. E., Sejnowski T. J.: A learning algorithm for Boltzmann machines. Cognitive Science 9 (1985), 147–169 DOI 10.1207/s15516709cog0901_7
[3] Aigner M.: Combinatorial Theory, Classics in Mathematics. Springer–Verlag, Berlin 1997 MR 1434477
[4] Amari S.: Information geometry on hierarchy of probability distributions. IEEE Trans. Inform. Theory 47 (2001), 1701–1711 DOI 10.1109/18.930911 | MR 1842511 | Zbl 0997.94009
[5] Amari S., Kurata, K., Nagaoka H.: Information geometry of Boltzmann machines. IEEE Trans. Neural Networks 3 (1992), 2, 260–271 DOI 10.1109/72.125867
[6] Ay N.: An information-geometric approach to a theory of pragmatic structuring. Ann. Probab. 30 (2002), 416–436 DOI 10.1214/aop/1020107773 | MR 1894113 | Zbl 1010.62007
[7] Ay N.: Locality of global stochastic interaction in directed acyclic networks. Neural Computation 14 (2002), 2959–2980 DOI 10.1162/089976602760805368 | Zbl 1079.68582
[8] Linsker R.: Self-organization in a perceptual network. IEEE Computer 21 (1988), 105–117 DOI 10.1109/2.36
[9] Matúš F., Ay N.: On maximization of the information divergence from an exponential family. In: Proc. WUPES’03 (J. Vejnarová, ed.), University of Economics, Prague 2003, pp. 199–204
[10] Shannon C. E.: A mathematical theory of communication. Bell System Tech. J. 27 (1948), 379–423, 623–656 DOI 10.1002/j.1538-7305.1948.tb01338.x | MR 0026286 | Zbl 1154.94303
[11] Tononi G., Sporns, O., Edelman G. M.: A measure for brain complexity: Relating functional segregation and integration in the nervous system. Proc. Nat. Acad. Sci. U. S. A. 91 (1994), 5033–5037 DOI 10.1073/pnas.91.11.5033
Search
Partner of
