Optimizing Workflow Data Footprint
Abstract
In this paper we examine the issue of optimizing disk usage and scheduling large-scale scientific workflows onto distributed resources where the workflows are data-intensive, requiring large amounts of data storage, and the resources have limited storage resources. Our approach is two-fold: we minimize the amount of space a workflow requires during execution by removing data files at runtime when they are no longer needed and we demonstrate that workflows may have to be restructured to reduce the overall data footprint of the workflow. We show the results of our data management and workflow restructuring solutions using a Laser Interferometer Gravitational-Wave Observatory (LIGO) application and an astronomy application, Montage, running on a large-scale production grid-the Open Science Grid. We show that although reducing the data footprint of Montage by 48% can be achieved with dynamic data cleanup techniques, LIGO Scientific Collaboration workflows require additional restructuring to achieve a 56% reduction in data space usage. We also examine the cost of the workflow restructuring in terms of the application's runtime.
Additional Information
© 2007 Hindawi Publishing Corporation. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. This work was supported by the National Science Foundation under the grant CNS 0615412. R. Sakellariou and H. Zhao would like to acknowledge partial support from the EU-funded CoreGrid Network of Excellence (grant FP6-004265) and the UK EPSRC grant GR/S67654/01. The authors also thank the Open Science Grid for resources used for the motivation of this work. K. Blackburn and D. Meyers were supported by the LIGO Laboratory and NSF grants PHY-0107417 and PHY-0326281. The work of D. Brown was supported by the LIGO Laboratory and NSF grant PHY-0601459. The work of S. Fairhurst was supported by the LIGO Laboratory and NSF grant PHY-0326281 and PHY-0200852. LIGO was constructed by the California Institute of Technology and the Massachusetts Institute of Technology and operates under cooperative agreement PHY-0107417. This paper has been assigned LIGO Document Number LIGO-P070017-00-Z. Montage was supported by the NASA Earth Sciences Technology Office Computing Technologies (ESTOCT) program under Cooperative Agreement Notice NCC 5-6261. This research was done using resources provided by the Open Science Grid, which is supported by the National Science Foundation and the U.S. Department of Energy's Office of Science.Attached Files
Published - 701609.pdf
Files
| Name | Size | Download all |
|---|---|---|
|
md5:59f9ab8e89bf5084e04a61e339ad7b5f
|
1.8 MB | Preview Download |
Additional details
- Eprint ID
- 86108
- Resolver ID
- CaltechAUTHORS:20180427-155708572
- CNS-0615412
- NSF
- FP6-004265
- CoreGrid Network of Excellence
- GR/S67654/01
- Engineering and Physical Sciences Research Council (EPSRC)
- LIGO Laboratory
- PHY-0107417
- NSF
- PHY-0326281
- NSF
- PHY-0601459
- NSF
- PHY-0326281
- NSF
- PHY-0200852
- NSF
- PHY-0107417
- NSF
- NCC 5-6261
- NASA
- Department of Energy (DOE)
- Created
-
2018-04-30Created from EPrint's datestamp field
- Updated
-
2021-11-15Created from EPrint's last_modified field
- Caltech groups
- Infrared Processing and Analysis Center (IPAC), LIGO
- Other Numbering System Name
- LIGO Document
- Other Numbering System Identifier
- P070017-00-Z