Iterative Amortized Policy Optimization
Abstract
Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control, enabling the estimation and sampling of high-value actions. From the variational inference perspective on RL, policy networks, when used with entropy or KL regularization, are a form of amortized optimization, optimizing network parameters rather than the policy distributions directly. However, direct amortized mappings can yield suboptimal policy estimates and restricted distributions, limiting performance and exploration. Given this perspective, we consider the more flexible class of iterative amortized optimizers. We demonstrate that the resulting technique, iterative amortized policy optimization, yields performance improvements over direct amortization on benchmark continuous control tasks.
Additional Information
JM acknowledges Scott Fujimoto for helpful discussions. This work was funded in part by NSF #1918839 and Beyond Limits. JM is currently employed by Google DeepMind. The authors declare no other competing interests related to this work.Additional details
- Eprint ID
- 118584
- Resolver ID
- CaltechAUTHORS:20221222-185354379
- NSF
- CCF-1918839
- Beyond Limits
- Created
-
2022-12-22Created from EPrint's datestamp field
- Updated
-
2022-12-22Created from EPrint's last_modified field
- Caltech groups
- Center for Autonomous Systems and Technologies (CAST)