Scipediacontent: Scipediacontent moved page Draft Content 481826966 to Dugre et al 2019a

2021-01-28T16:45:32Z

Scipediacontent moved page Draft Content 481826966 to Dugre et al 2019a

Scipediacontent: Created page with " == Abstract == In the past few years, neuroimaging has entered the Big Data era due to the joint increase in image resolution, data sharing, and study sizes. However, no par..."

2021-01-28T16:45:29Z

Created page with " == Abstract == In the past few years, neuroimaging has entered the Big Data era due to the joint increase in image resolution, data sharing, and study sizes. However, no par..."

New page

== Abstract ==

In the past few years, neuroimaging has entered the Big Data era due to the joint increase in image resolution, data sharing, and study sizes. However, no particular Big Data engines have emerged in this field, and several alternatives remain available. We compare two popular Big Data engines with Python APIs, Apache Spark and Dask, for their runtime performance in processing neuroimaging pipelines. Our evaluation uses two synthetic pipelines processing the 81GB BigBrain image, and a real pipeline processing anatomical data from more than 1,000 subjects. We benchmark these pipelines using various combinations of task durations, data sizes, and numbers of workers, deployed on an 8-node (8 cores ea.) compute cluster in Compute Canada's Arbutus cloud. We evaluate PySpark's RDD API against Dask's Bag, Delayed and Futures. Results show that despite slight differences between Spark and Dask, both engines perform comparably. However, Dask pipelines risk being limited by Python's GIL depending on task type and cluster configuration. In all cases, the major limiting factor was data transfer. While either engine is suitable for neuroimaging pipelines, more effort needs to be placed in reducing data transfer time.

Comment: 10 pages, 15 figures, 1 tables. To appear in the proceeding of the 14th WORKS Workshop on Topics in Workflows in Support of Large-Scale Science, 17 November 2019, Denver, CO, USA

== Original document ==

The different versions of the original document can be found in:

* [http://arxiv.org/abs/1907.13030 http://arxiv.org/abs/1907.13030]

* [http://arxiv.org/pdf/1907.13030 http://arxiv.org/pdf/1907.13030]

* [http://xplorestaging.ieee.org/ielx7/8937308/8943488/08943502.pdf?arnumber=8943502 http://xplorestaging.ieee.org/ielx7/8937308/8943488/08943502.pdf?arnumber=8943502],
: [http://dx.doi.org/10.1109/works49585.2019.00010 http://dx.doi.org/10.1109/works49585.2019.00010]

* [https://dblp.uni-trier.de/db/journals/corr/corr1907.html#abs-1907-13030 https://dblp.uni-trier.de/db/journals/corr/corr1907.html#abs-1907-13030],
: [https://arxiv.org/pdf/1907.13030.pdf https://arxiv.org/pdf/1907.13030.pdf],
: [https://arxiv.org/abs/1907.13030 https://arxiv.org/abs/1907.13030],
: [https://ui.adsabs.harvard.edu/abs/2019arXiv190713030D/abstract https://ui.adsabs.harvard.edu/abs/2019arXiv190713030D/abstract],
: [https://academic.microsoft.com/#/detail/3000137827 https://academic.microsoft.com/#/detail/3000137827]

← Older revision	Revision as of 16:45, 28 January 2021
(No difference)

Dugre et al 2019a - Revision history

Scipediacontent: Scipediacontent moved page Draft Content 481826966 to Dugre et al 2019a

Scipediacontent: Created page with " == Abstract == In the past few years, neuroimaging has entered the Big Data era due to the joint increase in image resolution, data sharing, and study sizes. However, no par..."