Scipediacontent: Scipediacontent moved page Draft Content 441602947 to Langius et al 2017a

2021-02-08T09:17:49Z

Scipediacontent moved page Draft Content 441602947 to Langius et al 2017a

Scipediacontent: Created page with " == Abstract == Distributed data processing systems are the standard means for large-scale data analysis in the Big Data field. These systems are based on processing pipeline..."

2021-02-08T09:17:46Z

Created page with " == Abstract == Distributed data processing systems are the standard means for large-scale data analysis in the Big Data field. These systems are based on processing pipeline..."

New page

== Abstract ==

Distributed data processing systems are the standard means for large-scale data analysis in the Big Data field. These systems are based on processing pipelines where the processing is done via a composition of multiple elements or steps. In current distributed data processing systems, the code and parameters that create the pipeline are set at design time, before the application starts processing any data. Any changes that have to be applied to the pipeline after it has been started, require the entire pipeline to be restarted. When a system needs to be operational 24/7 or has to respond in a timely fashion, restarting and having downtime is not acceptable. In this case, computing should be performed autonomously by the processing system that continuously takes the changes from the environment, and adjusts its processing steps, parameters, etc. on-the-fly. In this paper, we try to solve this problem by allowing changes to be made to a processing pipeline without restarting. We focus on two aspects of the problem: switching to another data source that is used as input, and changing the functional code and variables within the elements of a pipeline. Our system is built on top of Apache Spark, a framework widely used for distributed data processing.

== Original document ==

The different versions of the original document can be found in:

* [http://dx.doi.org/10.1109/iccac.2017.11 http://dx.doi.org/10.1109/iccac.2017.11]

* [http://resolver.tudelft.nl/uuid:a35d75d0-5adf-4afd-974b-205575f6deb1 http://resolver.tudelft.nl/uuid:a35d75d0-5adf-4afd-974b-205575f6deb1]

* [https://www.rug.nl/research/portal/en/publications/runtime-modifications-of-spark-data-processing-pipelines(d09549b2-f5d2-40b9-a093-701b5abb4505).html https://www.rug.nl/research/portal/en/publications/runtime-modifications-of-spark-data-processing-pipelines(d09549b2-f5d2-40b9-a093-701b5abb4505).html]

* [http://xplorestaging.ieee.org/ielx7/8063635/8064040/08064052.pdf?arnumber=8064052 http://xplorestaging.ieee.org/ielx7/8063635/8064040/08064052.pdf?arnumber=8064052],
: [http://dx.doi.org/10.1109/iccac.2017.11 http://dx.doi.org/10.1109/iccac.2017.11]

* [https://www.rug.nl/research/portal/en/publications/runtime-modifications-of-spark-data-processing-pipelines(d09549b2-f5d2-40b9-a093-701b5abb4505).html https://www.rug.nl/research/portal/en/publications/runtime-modifications-of-spark-data-processing-pipelines(d09549b2-f5d2-40b9-a093-701b5abb4505).html],
: [https://repository.tudelft.nl/view/tno/uuid:a35d75d0-5adf-4afd-974b-205575f6deb1 https://repository.tudelft.nl/view/tno/uuid:a35d75d0-5adf-4afd-974b-205575f6deb1],
: [https://www.narcis.nl/publication/RecordID/oai%3Atudelft.nl%3Auuid%3Aa35d75d0-5adf-4afd-974b-205575f6deb1 https://www.narcis.nl/publication/RecordID/oai%3Atudelft.nl%3Auuid%3Aa35d75d0-5adf-4afd-974b-205575f6deb1],
: [https://dblp.uni-trier.de/db/conf/iccac/iccac2017.html#LazovikMALL17 https://dblp.uni-trier.de/db/conf/iccac/iccac2017.html#LazovikMALL17],
: [https://ieeexplore.ieee.org/document/8064052 https://ieeexplore.ieee.org/document/8064052],
: [https://doi.org/10.1109/ICCAC.2017.11 https://doi.org/10.1109/ICCAC.2017.11],
: [https://academic.microsoft.com/#/detail/2761235021 https://academic.microsoft.com/#/detail/2761235021]

← Older revision	Revision as of 09:17, 8 February 2021
(No difference)

Langius et al 2017a - Revision history

Scipediacontent: Scipediacontent moved page Draft Content 441602947 to Langius et al 2017a

Scipediacontent: Created page with " == Abstract == Distributed data processing systems are the standard means for large-scale data analysis in the Big Data field. These systems are based on processing pipeline..."