Scipediacontent: Scipediacontent moved page Draft Content 510035742 to Atkinson et al 2010a

2021-02-03T14:52:00Z

Scipediacontent moved page Draft Content 510035742 to Atkinson et al 2010a

Scipediacontent: Created page with " == Abstract == Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multi-disciplinary expertise and large-scale computat..."

2021-02-03T14:51:58Z

Created page with " == Abstract == Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multi-disciplinary expertise and large-scale computat..."

New page

== Abstract ==

Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multi-disciplinary expertise and large-scale computational experiments. These experiments usually involve large amounts of data that are located in distributed data repositories running various software systems, and managed by different organisations. A common strategy to make the experiments more manageable is executing the processing steps as a workflow. In this paper, we look into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams. We model the distributed computation as a directed acyclic graph where the nodes represent the processing elements that incrementally implement specific subtasks. The processing elements are connected in a pipelined streaming manner, which allows task executions to overlap. We further optimise the execution by splitting pipelines across processes and by introducing extra parallel streams. We identify performance metrics and design a measurement tool to evaluate each enactment. We conducted experiments to evaluate our optimisation strategies with a real world problem in the Life Sciences---EURExpress-II. The paper presents our distributed data-handling model, the optimisation and instrumentation strategies and the evaluation experiments. We demonstrate linear speed up and argue that this use of data-streaming to enable both overlapped pipeline and parallelised enactment is a generally applicable optimisation strategy.

== Original document ==

The different versions of the original document can be found in:

* [https://www.cct.lsu.edu/~kosar/didc10/papers/didc03.pdf https://www.cct.lsu.edu/~kosar/didc10/papers/didc03.pdf]

* [https://www.cct.lsu.edu/~kosar/didc10/papers/didc03.pdf https://www.cct.lsu.edu/~kosar/didc10/papers/didc03.pdf],
: [https://dblp.uni-trier.de/db/conf/hpdc/hpdc2010.html#LiewAHH10 https://dblp.uni-trier.de/db/conf/hpdc/hpdc2010.html#LiewAHH10],
: [https://dl.acm.org/citation.cfm?doid=1851476.1851583 https://dl.acm.org/citation.cfm?doid=1851476.1851583],
: [http://portal.acm.org/citation.cfm?doid=1851476.1851583 http://portal.acm.org/citation.cfm?doid=1851476.1851583],
: [http://www.research.ed.ac.uk/portal/en/publications/towards-optimising-distributed-data-streaming-graphs-using-parallel-streams(3b90d9fa-9ff6-4e02-9178-e3ade4b1b2ec)/export.html http://www.research.ed.ac.uk/portal/en/publications/towards-optimising-distributed-data-streaming-graphs-using-parallel-streams(3b90d9fa-9ff6-4e02-9178-e3ade4b1b2ec)/export.html],
: [https://core.ac.uk/display/102463250 https://core.ac.uk/display/102463250],
: [https://academic.microsoft.com/#/detail/2094271094 https://academic.microsoft.com/#/detail/2094271094]

* [http://dl.acm.org/ft_gateway.cfm?id=1851583&ftid=840361&dwn=1 http://dl.acm.org/ft_gateway.cfm?id=1851583&ftid=840361&dwn=1],
: [http://dx.doi.org/10.1145/1851476.1851583 http://dx.doi.org/10.1145/1851476.1851583]

* [ ]

← Older revision	Revision as of 14:52, 3 February 2021
(No difference)

Atkinson et al 2010a - Revision history

Scipediacontent: Scipediacontent moved page Draft Content 510035742 to Atkinson et al 2010a

Scipediacontent: Created page with " == Abstract == Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multi-disciplinary expertise and large-scale computat..."