Understanding and Improving Reliability of Inference Dynamics in Deep Neural Networks

Citation

Huang, Yujia (2024) Understanding and Improving Reliability of Inference Dynamics in Deep Neural Networks. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/77cc-6z86. https://resolver.caltech.edu/CaltechTHESIS:03052024-025023714

Abstract

Reliability is a crucial aspect for the successful deployment of deep learning systems across various domains. In generative modeling, it is essential to create content that adheres to specific rules. In the field of control, ensuring that robots operate safely without falling or entering hazardous areas is paramount. Similarly, in visual perception, the robustness of perception results against perturbations are vital.

In this thesis, we explore the reliability of inference dynamics in deep neural networks such as ResNet, neural Ordinary Differential Equations (ODEs), and diffusion models. We begin by examining the inference dynamics in standard networks with a discrete sequence of hidden layers, applying self-consistency and local Lipschitz bounds to enhance robustness against input perturbations. Our exploration then extends to neural ODEs, where the neural network specifies a vector field that continuously transforms the state. We employ forward invariance to achieve robustness, marking the first instance of training neural ODE policies with non-vacuous certified guarantees. The focus shifts next to diffusion models and their inference processes, particularly in adhering to symbolic constraints. For this, we introduce a novel sampling algorithm inspired by stochastic control principles. This algorithm not only guides these models in generating rule-specific content but also sets a new benchmark in symbolic music generation. Our work offers a cohesive understanding of inference dynamics in various deep learning architectures and propose new algorithms to significantly improve their reliability.

Thesis Files