Latest revision as of 17:10, 21 January 2021

Abstract

Accepted manuscript version. The final publication is available at Springer via <a href=http://dx.doi.org/10.1007/978-3-319-24462-4_22>http://dx.doi.org/10.1007/978-3-319-24462-4_22</a>. Biological data analysis is typically implemented using a deep pipeline that combines a wide array of tools and databases. These pipelines must scale to very large datasets, and consequently require parallel and distributed computing. It is therefore important to choose a hardware platform and underlying data management and processing systems well suited for processing large datasets. There are many infrastructure systems for such data-intensive computing. However, in our experience, most biological data analysis pipelines do not leverage these systems. We give an overview of data-intensive computing infrastructure systems, and describe how we have leveraged these for: (i) scalable fault-tolerant computing for large-scale biological data; (ii) incremental updates to reduce the resource usage required to update large-scale compendium; and (iii) interactive data analysis and exploration. We provide lessons learned and describe problems we have encountered during development and deployment. We also provide a literature survey on the use of data-intensive computing systems for biological data processing. Our results show how unmodified biological data analysis tools can benefit from infrastructure systems for data-intensive computing.

Original document

The different versions of the original document can be found in:

• https://hdl.handle.net/10037/8816

• https://munin.uit.no/bitstream/10037/8816/2/article.pdf

• http://link.springer.com/content/pdf/10.1007/978-3-319-24462-4_22,

http://dx.doi.org/10.1007/978-3-319-24462-4_22 under the license http://www.springer.com/tdm

• https://link.springer.com/chapter/10.1007/978-3-319-24462-4_22,

https://core.ac.uk/display/141562029,

https://munin.uit.no/bitstream/10037/8816/2/article.pdf,

https://munin.uit.no/handle/10037/8816,

https://dblp.uni-trier.de/db/conf/cibb/cibb2014.html#BongoPE14,

https://rd.springer.com/chapter/10.1007/978-3-319-24462-4_22,

https://academic.microsoft.com/#/detail/2296514683