Bongo et al 2015a - Revision history

Scipediacontent at 15:10, 21 January 2021

2021-01-21T15:10:23Z

Scipediacontent: Scipediacontent moved page Draft Content 294469788 to Bongo et al 2015a

2020-10-14T12:01:56Z

Scipediacontent moved page Draft Content 294469788 to Bongo et al 2015a

Scipediacontent: Created page with " == Abstract == Accepted manuscript version. The final publication is available at Springer via http://dx.doi.org/10.10..."

2020-10-14T12:01:54Z

Created page with " == Abstract == Accepted manuscript version. The final publication is available at Springer via <a href=http://dx.doi.org/10.1007/978-3-319-24462-4_22>http://dx.doi.org/10.10..."

New page

== Abstract ==

Accepted manuscript version. The final publication is available at Springer via <a href=http://dx.doi.org/10.1007/978-3-319-24462-4_22>http://dx.doi.org/10.1007/978-3-319-24462-4_22</a>. Biological data analysis is typically implemented using a deep pipeline that combines a wide array of tools and databases. These pipelines must scale to very large datasets, and consequently require parallel and distributed computing. It is therefore important to choose a hardware platform and underlying data management and processing systems well suited for processing large datasets. There are many infrastructure systems for such data-intensive computing. However, in our experience, most biological data analysis pipelines do not leverage these systems. We give an overview of data-intensive computing infrastructure systems, and describe how we have leveraged these for: (i) scalable fault-tolerant computing for large-scale biological data; (ii) incremental updates to reduce the resource usage required to update large-scale compendium; and (iii) interactive data analysis and exploration. We provide lessons learned and describe problems we have encountered during development and deployment. We also provide a literature survey on the use of data-intensive computing systems for biological data processing. Our results show how unmodified biological data analysis tools can benefit from infrastructure systems for data-intensive computing.

Document type: Part of book or chapter of book

== Full document ==
<pdf>Media:Draft_Content_294469788-beopen445-5616-document.pdf</pdf>

== Original document ==

The different versions of the original document can be found in:

* [http://hdl.handle.net/10037/8816 http://hdl.handle.net/10037/8816]

* [https://hdl.handle.net/10037/8816 https://hdl.handle.net/10037/8816]

* [https://munin.uit.no/bitstream/10037/8816/2/article.pdf https://munin.uit.no/bitstream/10037/8816/2/article.pdf]

@@ Line 3: / Line 3: @@
 Accepted manuscript version. The final publication is available at Springer via <a href=http://dx.doi.org/10.1007/978-3-319-24462-4_22>http://dx.doi.org/10.1007/978-3-319-24462-4_22</a>. Biological data analysis is typically implemented using a deep pipeline that combines a wide array of tools and databases. These pipelines must scale to very large datasets, and consequently require parallel and distributed computing. It is therefore important to choose a hardware platform and underlying data management and processing systems well suited for processing large datasets. There are many infrastructure systems for such data-intensive computing. However, in our experience, most biological data analysis pipelines do not leverage these systems. We give an overview of data-intensive computing infrastructure systems, and describe how we have leveraged these for: (i) scalable fault-tolerant computing for large-scale biological data; (ii) incremental updates to reduce the resource usage required to update large-scale compendium; and (iii) interactive data analysis and exploration. We provide lessons learned and describe problems we have encountered during development and deployment. We also provide a literature survey on the use of data-intensive computing systems for biological data processing. Our results show how unmodified biological data analysis tools can benefit from infrastructure systems for data-intensive computing.
@@ Line 13: / Line 8: @@
 The different versions of the original document can be found in:
 * [https://hdl.handle.net/10037/8816 https://hdl.handle.net/10037/8816]
 * [https://munin.uit.no/bitstream/10037/8816/2/article.pdf https://munin.uit.no/bitstream/10037/8816/2/article.pdf]

← Older revision	Revision as of 12:01, 14 October 2020
(No difference)