Instantaneous Archives
- Creators
- Landry, Walter
- Monkewitz, Serge
Abstract
IRSA, the NASA/IPAC Infrared Science Archive, is one of the largest and busiest astronomy archives in the world. In the past, our main emphasis was on making new data and new capabilities available. With the widespread implementation of Virtual Observatory protocols, there are a number of useful tools that can quickly and easily perform insightful, sophisticated queries from archives around the world (Taylor et al. 2019; Astropy Collaboration et al. 2013). The queries, if not handled quickly, can easily overwhelm the site and interfere with other users. In addition, reducing latency below the point of human perception enables more interactive and exploratory science. This paper discusses how we are re-architecting our query pipeline to eliminate process, file system, and database connection overheads. Taken together, these improvements delivered radical, order of magnitude improvements in latency and throughput.
Additional Information
© 2019 Astronomical Society of the Pacific. We would like to thank Mark O'Dell, Angela Zhang, Scott Terek, and Gilles Landais for their assistance.Additional details
- Eprint ID
- 100120
- Resolver ID
- CaltechAUTHORS:20191202-075406415
- Created
-
2019-12-02Created from EPrint's datestamp field
- Updated
-
2020-11-18Created from EPrint's last_modified field
- Caltech groups
- Infrared Processing and Analysis Center (IPAC)
- Series Name
- Astronomical Society of the Pacific Conference Series
- Series Volume or Issue Number
- 521