Cloudflow – A Framework for MapReduce Pipeline Development in Biomedical Research

Abstract

- The data-driven parallelization framework Hadoop MapReduce allows analysing large data sets in a scalable way. Since the development of MapReduce programs can be a time-intensive and challenging task, the application and usage of Hadoop in Biomedical Research is still limited. Here we present Cloudflow, a high-level framework to hide the implementation details of Hadoop and to provide a set of building blocks to create biomedical pipelines in a more intuitive way. We demonstrate the benefit of Cloudflow on three different genetic use cases. It will be shown how the framework can be combined with the Hadoop workflow system Cloudgene and the cloud orchestration platform CloudMan to provide Hadoop pipelines as a service to everyone. The framework is open source and free available at https://github.com/genepi/cloudflow.

Document type: Conference object

Full document

Original document

The different versions of the original document can be found in:

• http://dx.doi.org/10.6084/m9.figshare.1424739 under the license https://creativecommons.org/licenses/by/4.0/legalcode

• http://dx.doi.org/10.6084/m9.figshare.1424739.v1 under the license https://creativecommons.org/licenses/by/4.0/legalcode

• http://fulir.irb.hr/1988/1/Cloudflow%20-%20A%20Framework%20for%20MapReduce%20Pipile%20Development%20in%20Biomedical%20Research.pdf

• http://fulir.irb.hr/1988,

http://mipro-proceedings.com,

http://fulir.irb.hr/1988/1/Cloudflow%20-%20A%20Framework%20for%20MapReduce%20Pipile%20Development%20in%20Biomedical%20Research.pdf

• http://xplorestaging.ieee.org/ielx7/7153596/7160221/07160259.pdf?arnumber=7160259,

http://dx.doi.org/10.1109/mipro.2015.7160259

• https://dblp.uni-trier.de/db/conf/mipro/mipro2015.html#ForerAWDSKS15,

http://ieeexplore.ieee.org/document/7160259,

https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/2080825/CloudflowSUBMITTED.pdf,

https://doi.org/10.1109/MIPRO.2015.7160259,

http://fulir.irb.hr/1988,

https://academic.microsoft.com/#/detail/1605032987

DOIS: 10.6084/m9.figshare.1424739.v1 10.1109/mipro.2015.7160259 10.6084/m9.figshare.1424739