Astronomical Image Processing at Scale With Pegasus and Montage

Abstract

Image processing at scale is a powerful tool for creating new data sets and integrating them with existing data sets and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involves multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from failures, and monitoring of performance. In this focus demo we demonstrate how the Pegasus Workflow Manager Python API manages image processing to create mosaics with the Montage Image Mosaic engine. Since 2001, Pegasus has been developed and maintained at USC/ISI. Montage was in fact one of the first applications used to design Pegasus and optimize its performance. Pegasus has since found application in many areas of science. LIGO exploited it in making discoveries of black holes. The Vera C. Rubin Observatory used it to compare the cost and performance of processing images on cloud platforms. While these are examples of projects at large scale, small team investigations on local clusters of machines can benefit from Pegasus as well.

Files

Additional details