Abstract

Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multi-disciplinary expertise and large-scale computational experiments. These experiments usually involve large amounts of data that are located in distributed data repositories running various software systems, and managed by different organisations. A common strategy to make the experiments more manageable is executing the processing steps as a workflow. In this paper, we look into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams. We model the distributed computation as a directed acyclic graph where the nodes represent the processing elements that incrementally implement specific subtasks. The processing elements are connected in a pipelined streaming manner, which allows task executions to overlap. We further optimise the execution by splitting pipelines across processes and by introducing extra parallel streams. We identify performance metrics and design a measurement tool to evaluate each enactment. We conducted experiments to evaluate our optimisation strategies with a real world problem in the Life Sciences---EURExpress-II. The paper presents our distributed data-handling model, the optimisation and instrumentation strategies and the evaluation experiments. We demonstrate linear speed up and argue that this use of data-streaming to enable both overlapped pipeline and parallelised enactment is a generally applicable optimisation strategy.


Original document

The different versions of the original document can be found in:

https://dblp.uni-trier.de/db/conf/hpdc/hpdc2010.html#LiewAHH10,
https://dl.acm.org/citation.cfm?doid=1851476.1851583,
http://portal.acm.org/citation.cfm?doid=1851476.1851583,
http://www.research.ed.ac.uk/portal/en/publications/towards-optimising-distributed-data-streaming-graphs-using-parallel-streams(3b90d9fa-9ff6-4e02-9178-e3ade4b1b2ec)/export.html,
https://core.ac.uk/display/102463250,
https://academic.microsoft.com/#/detail/2094271094
http://dx.doi.org/10.1145/1851476.1851583
  • [ ]
Back to Top

Document information

Published on 01/01/2010

Volume 2010, 2010
DOI: 10.1145/1851476.1851583
Licence: CC BY-NC-SA license

Document Score

0

Views 0
Recommendations 0

Share this document

Keywords

claim authorship

Are you one of the authors of this document?