Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published December 1991 | Published
Journal Article Open

Temporal evolution of generalization during learning in linear networks

Abstract

We study generalization in a simple framework of feedforward linear networks with n inputs and n outputs, trained from examples by gradient descent on the usual quadratic error function. We derive analytical results on the behavior of the validation function corresponding to the LMS error function calculated on a set of validation patterns. We show that the behavior of the validation function depends critically on the initial conditions and on the characteristics of the noise. Under certain simple assumptions, if the initial weights are sufficiently small, the validation function has a unique minimum corresponding to an optimal stopping time for training for which simple bounds can be calculated. There exists also situations where the validation function can have more complicated and somewhat unexpected behavior such as multiple local minima (at most n) of variable depth and long but finite plateau effects. Additional results and possible extensions are briefly discussed.

Additional Information

© 1991 Massachusetts Institute of Technology. Received 1 February 1991; accepted 13 April 1991. Posted Online March 13, 2008. This work is in part supported by grants from the Office of Naval Research and the McDonnell-Pew foundation to P. B. We would like to thank Yosi Rinott for useful discussions.

Attached Files

Published - BALnc91b.pdf

Files

BALnc91b.pdf
Files (658.5 kB)
Name Size Download all
md5:d87ac67fa4d188129a10b5efaf2173a4
658.5 kB Preview Download

Additional details

Created:
August 20, 2023
Modified:
October 18, 2023