On the structure of decentralized controllers in networked MDPs

Abstract

This work considers the problem of constructing optimal decentralized controllers for networked Markov Decision Processes (MDPs). A number of subsystems, each modeled as an MDP, are dynamically coupled through a network and affect one another with delay. Each subsystem has a corresponding controller which has perfect knowledge of the local state and may communicate its state to the other controllers over a noise-free channel with fixed delay. The problem is framed in terms of a coordinator with access to the shared information of the controllers, leading to a dynamic programming problem to calculate optimal policies. Further, under the criteria of partial nestedness the dynamic program exhibits additional factored structure, leading to more efficient synthesis of the optimal policy.

Additional details