Published October 2022
| Published
Journal Article
Open
Mean-field limits of trained weights in deep learning: A dynamical systems perspective
Abstract
Training a residual neural network with L^2 regularization on weights and biases is equivalent to minimizing a discrete least action principle and to controlling a discrete Hamiltonian system representing the propagation of input data across layers. The kernel/feature map analysis of this Hamiltonian system suggests a mean-field limit for trained weights and biases as the number of data points goes to infinity. The purpose of this paper is to investigate this mean-field limit and illustrate its existence through numerical experiments and analysis (for simple kernels).
Additional Information
© 2022 The Author(s). Volume 15, Special Issue dedicated to Robert Schaback on the occasion of his 75th birthday (2022). Parts of this work were done when B. H. was a Marie Curie fellow at Imperial College London. B. H. thanks the European Commission for funding through the Marie Curie fellowship STALDYS-792919 (Statistical Learning for Dynamical Systems). H. O. gratefully acknowledges support by the Air Force Office of Scientific Research under award number FA9550-18-1-0271 (Games for Computation and Learning). Code. The Python codes of all numerical experiments in this paper are at https://github.com/alexsm98/Mean-field-limit-code-.gitAttached Files
Published - 12_smirnov.pdf
Files
12_smirnov.pdf
Files
(1.7 MB)
Name | Size | Download all |
---|---|---|
md5:c7272ecb7d045fbf00227edacc2b0ae9
|
1.7 MB | Preview Download |
Additional details
- Eprint ID
- 121706
- Resolver ID
- CaltechAUTHORS:20230603-041912328
- Marie Curie Fellowship
- 792919
- Air Force Office of Scientific Research (AFOSR)
- FA9550-18-1-0271
- Created
-
2023-06-05Created from EPrint's datestamp field
- Updated
-
2023-06-05Created from EPrint's last_modified field