Scalability and Efficiency in Distributed Big Data Architectures: A Comparative Study

Abstract

With the rapid expansion of the size of data, there is a need for the development of scalable and efficient architectures for large scale data processing. This research conducts a comparative analysis between the performance, scalability and efficiency of the Apache Hadoop, Apache Spark, Apache Flink, and Google Bigtable big data frameworks. Finally, the experimental results indicate that the Apache Spark is faster in execution times by 3.5× than Hadoop, and the Apache Flink achieves 40% lower latency on real time analytics than Spark. Google Bigtable had good throughput at 5 million queries a second, but it was not flexible to computationally intense processes. Furthermore, this study examined the application of the machine learning and blockchain technologies in the implementation of the distributed systems for the unified backend that incorporates processing efficiency improvement by 25% and data integrity with the added computational overhead of 12%. The research demonstrates that Flink is most suitable for real time data streams, spark is the best tool for iterative workloads and bigtable is the most appropriate for structured high throughput storage. Nevertheless, questions remain on how to scale in the extreme workload case and balance security with performance. Finally, future research will focus on hybrid architectures that enable high speed and security performance for the next generation big data applications.