Scipediacontent: Scipediacontent moved page Draft Content 481691223 to Deelman et al 2015a

2021-01-28T22:19:12Z

Scipediacontent moved page Draft Content 481691223 to Deelman et al 2015a

Scipediacontent: Created page with " == Abstract == Scientific workflows are often used to automate large-scale data analysis pipelines on clusters, grids, and clouds. However, because workflows can be extremel..."

2021-01-28T22:19:08Z

Created page with " == Abstract == Scientific workflows are often used to automate large-scale data analysis pipelines on clusters, grids, and clouds. However, because workflows can be extremel..."

New page

== Abstract ==

Scientific workflows are often used to automate large-scale data analysis pipelines on clusters, grids, and clouds. However, because workflows can be extremely data-intensive, and are often executed on shared resources, it is critical to be able to limit or minimize the amount of disk space that workflows use on shared storage systems. This paper proposes a novel and simple approach that constrains the amount of storage space used by a workflow by inserting data cleanup tasks into the workflow task graph. Unlike previous solutions, the proposed approach provides guaranteed limits on disk usage, requires no new functionality in the underlying workflow scheduler, and does not require estimates of task runtimes. Experimental results show that this algorithm significantly reduces the number of cleanup tasks added to a workflow and yields better workflow makespans than the strategy currently used by the Pegasus Workflow Management System.

== Original document ==

The different versions of the original document can be found in:

* [http://raiith.iith.ac.in/1977/1/1977_raiith_ieee_07019861.pdf http://raiith.iith.ac.in/1977/1/1977_raiith_ieee_07019861.pdf]

* [http://xplorestaging.ieee.org/ielx7/7015729/7019851/07019861.pdf?arnumber=7019861 http://xplorestaging.ieee.org/ielx7/7015729/7019851/07019861.pdf?arnumber=7019861],
: [http://dx.doi.org/10.1109/works.2014.8 http://dx.doi.org/10.1109/works.2014.8]

* [https://dblp.uni-trier.de/db/conf/sc/works2014.html#SrinivasanJSVD14 https://dblp.uni-trier.de/db/conf/sc/works2014.html#SrinivasanJSVD14],
: [http://raiith.iith.ac.in/1977 http://raiith.iith.ac.in/1977],
: [https://ieeexplore.ieee.org/document/7019861 https://ieeexplore.ieee.org/document/7019861],
: [https://academic.microsoft.com/#/detail/2105512718 https://academic.microsoft.com/#/detail/2105512718]

← Older revision	Revision as of 22:19, 28 January 2021
(No difference)

Deelman et al 2015a - Revision history

Scipediacontent: Scipediacontent moved page Draft Content 481691223 to Deelman et al 2015a

Scipediacontent: Created page with " == Abstract == Scientific workflows are often used to automate large-scale data analysis pipelines on clusters, grids, and clouds. However, because workflows can be extremel..."