Abstract

In this paper, we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas. We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark dataframes and Scala application programming interface. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.


Original document

The different versions of the original document can be found in:

http://dx.doi.org/10.1109/bigdata.2016.7841068
https://academic.microsoft.com/#/detail/2584392848
Back to Top

Document information

Published on 01/01/2019

Volume 2019, 2019
DOI: 10.1109/bigdata.2016.7841068
Licence: CC BY-NC-SA license

Document Score

0

Views 2
Recommendations 0

Share this document

claim authorship

Are you one of the authors of this document?