APS March Meeting 2024, Monday–Friday, March 4–8, 2024; Minneapolis & Virtual
Session S28: Statistical Physics Meets Machine Learning I, 8:00 AM–11:00 AM, Thursday, March 7, 2024, Room 101I
Abstract: S28.00005 : Criticality from the functional development of a learning machine
Time: 9:12 AM–9:24 AM
Presenter: Ting-Kuo Lee (National Tsing Hua University)
Author: Ting-Kuo Lee (National Tsing Hua University)
With increasing integrations of artificial intelligence in our everyday lives, the understanding and accountability of the solutions provided by machines are ever more important. This parallels the study of animal brains and biological neural networks. Instead of tracing out microscopic details of the neural networks that produce specific functionality, we aim for a macroscopic understanding of the evolution and transitions of the collective states of the neural systems leading to the fulfillment of the required functions. To this end, we apply the statistical modeling approach to the internal states of a pattern-recognizing neural network and characterize the training process of the network system with the evolution of the thermodynamic properties of the models. An overall trend of tending to a critical state of the mapped model over the training process is found with the ensemble of machines we considered. The entropy of the system is also estimated using the model and shows a non-monotonic variation with a minimum before the network is optimally trained as well as a plateau afterward. Using the relevance heat maps obtained with the deep-Taylor decomposition, which attributes output decisions of the network to relevant pixels of the input, we also find a maximum of the contrast or sharpness of the heat maps over the training. These findings support the presence of distinct stages of the learning process and improve our understanding of machine learning in neural networks.