Scipediacontent: Scipediacontent moved page Draft Content 215777622 to O'Leary et al 2018a

2021-02-01T22:16:58Z

Scipediacontent moved page Draft Content 215777622 to O'Leary et al 2018a

Scipediacontent: Created page with " == Abstract == dvances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increa..."

2021-02-01T22:16:53Z

Created page with " == Abstract == dvances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increa..."

New page

== Abstract ==

dvances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increases in the three Vs (Volume, Velocity, and Variety) of experimental data and the scale of computational tasks produced the demand for new real-time processing systems at experimental facilities. Recently, this demand was addressed by the Spark-MPI approach connecting the Spark data-intensive platform with the MPI high-performance framework. In contrast with existing data management and analytics systems, Spark introduced a new middleware based on resilient distributed datasets (RDDs), which decoupled various data sources from high-level processing algorithms. The RDD middleware significantly advanced the scope of data-intensive applications, spreading from SQL queries to machine learning to graph processing. Spark-MPI further extended the Spark ecosystem with the MPI applications using the Process Management Interface. The paper explores this integrated platform within the context of online ptychographic and tomographic reconstruction pipelines.

Comment: New York Scientific Data Summit, August 6-9, 2017

== Original document ==

The different versions of the original document can be found in:

* [http://arxiv.org/abs/1805.04886 http://arxiv.org/abs/1805.04886]

* [http://arxiv.org/pdf/1805.04886 http://arxiv.org/pdf/1805.04886]

* [http://dx.doi.org/10.1109/nysds.2017.8085039 http://dx.doi.org/10.1109/nysds.2017.8085039]

* [http://xplorestaging.ieee.org/ielx7/8065814/8085031/08085039.pdf?arnumber=8085039 http://xplorestaging.ieee.org/ielx7/8065814/8085031/08085039.pdf?arnumber=8085039],
: [http://dx.doi.org/10.1109/nysds.2017.8085039 http://dx.doi.org/10.1109/nysds.2017.8085039]

* [https://dblp.uni-trier.de/db/journals/corr/corr1805.html#abs-1805-04886 https://dblp.uni-trier.de/db/journals/corr/corr1805.html#abs-1805-04886],
: [https://ui.adsabs.harvard.edu/abs/2018arXiv180504886M/abstract https://ui.adsabs.harvard.edu/abs/2018arXiv180504886M/abstract],
: [https://ieeexplore.ieee.org/document/8085039 https://ieeexplore.ieee.org/document/8085039],
: [https://academic.microsoft.com/#/detail/2963747422 https://academic.microsoft.com/#/detail/2963747422]

← Older revision	Revision as of 22:16, 1 February 2021
(No difference)

O'Leary et al 2018a - Revision history

Scipediacontent: Scipediacontent moved page Draft Content 215777622 to O'Leary et al 2018a

Scipediacontent: Created page with " == Abstract == dvances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increa..."