Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published August 2020 | Supplemental Material + Published + Submitted
Journal Article Open

Orthogonal Gradient Descent for Continual Learning

Abstract

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.

Additional Information

© 2020 by the author(s). The authors would like to thank Dilan Gorur, Jonathan Schwarz, Jiachen Yang, and Yee Whye Teh for the comments and discussions.

Attached Files

Published - farajtabar20a.pdf

Submitted - 1910.07104.pdf

Supplemental Material - farajtabar20a-supp.pdf

Files

1910.07104.pdf
Files (1.7 MB)
Name Size Download all
md5:08be01f1299a6743b23540c48fc0843c
555.8 kB Preview Download
md5:c900a81e126697c51ec2757a713e5662
839.0 kB Preview Download
md5:fdaf370dc2f1625931ecaec244b28cfe
268.1 kB Preview Download

Additional details

Created:
August 19, 2023
Modified:
October 20, 2023