Article section

Article title
Contributor
Abstract
About article
Citation style
DOI
Copyright
Open access
References
Research Article

Enhancing Data Pipeline Efficiency Using Cloud-Based Big Data Technologies: A Comparative Analysis of AWS and Microsoft Azure

Authors

  • Olawumi Oladimeji Austin Peay state university, Clarksville, Tennessee, USA

Abstract

This study conducts a comprehensive comparative analysis of data pipeline efficiency between Amazon Web Services (AWS) Glue and Microsoft Azure Data Factory, two leading cloud-based big data technologies. As organizations increasingly rely on data-driven decision-making, optimizing data pipeline performance is crucial for processing large volumes of information from diverse sources. The research evaluates AWS Glue and Azure Data Factory based on key metrics such as processing speed, scalability, cost efficiency, and fault tolerance, using synthetic datasets ranging from 10GB to 500GB. The results indicate that AWS Glue consistently outperforms Azure Data Factory in processing speed and scalability, particularly for larger data sets, while Azure Data Factory offers greater cost efficiency for smaller workloads. Additionally, AWS Glue demonstrated superior fault tolerance, recovering more quickly from simulated errors compared to Azure Data Factory. These findings provide valuable insights for businesses and data professionals seeking to select the most suitable cloud platform for efficient data pipeline management. This study contributes to the growing body of knowledge on cloud-based big data technologies by offering an up-to-date evaluation of AWS and Azure's data pipeline efficiency, helping organizations optimize their big data processing strategies.

Article information

Journal

Journal of Multidisciplinary Research and Innovation

Volume (Issue)

2 (1)

Pages

11-19

Published

2023-10-11

How to Cite

Oladimeji, O. (2023). Enhancing Data Pipeline Efficiency Using Cloud-Based Big Data Technologies: A Comparative Analysis of AWS and Microsoft Azure. Journal of Multidisciplinary Research and Innovation, 2(1), 11-19. https://doi.org/10.70560/n43nvk83

References

Abduljabbar, Z., Omar, M., & Maabreh, M. (2020). The Role of Cloud Computing in Big Data Analytics. Journal of Cloud Computing, 9(3), 1-15. https://doi.org/10.1186/s13677-020-00178-0

Banerjee, S., & Roy, A. (2023). A Comparative Study of Azure Data Factory and AWS Glue in Big Data Processing. International Journal of Cloud Applications, 17(2), 89-102. https://doi.org/10.1016/j.ijca.2023.102234

Bhandari, A., & Sharma, P. (2023). Optimizing Big Data Pipelines in Cloud Environments. IEEE Transactions on Cloud Computing, 11(2), 239-251. https://doi.org/10.1109/TCC.2023.3245678

Chen, X., Li, Y., & Zhang, L. (2023). Enhancing Data Pipeline Efficiency in Cloud Environments: A Case Study on Azure Data Factory. Journal of Big Data Technologies, 14(1), 45-58. https://doi.org/10.1007/s41060-023-00190-y

Chen, Y., & Zhang, L. (2022). A Comparative Study of AWS and Azure in Big Data Processing. International Journal of Cloud Computing, 16(1), 112-126. https://doi.org/10.1504/IJCC.2022.10041869

Fronzetti Colladon, A., & Remondi, E. (2021). AWS Glue: A Framework for Data Integration and Analytics in Cloud Environments. Journal of Cloud Computing, 10(1), 67-78. https://doi.org/10.1186/s13677-021-00244-1

Gupta, A., Mohanty, R., & Sharma, P. (2021). Cloud-Based Data Pipelines: Opportunities and Challenges. Journal of Cloud Research, 9(4), 243-257. https://doi.org/10.1007/s13677-021-00302-x

Jain, V., Singh, K., & Roy, S. (2022). Evaluating the Efficiency of AWS Glue for Big Data Analytics. IEEE Transactions on Cloud Computing, 10(3), 285-298. https://doi.org/10.1109/TCC.2022.3184458

Kumar, S., & Singh, P. (2022). Enhancing Data Pipeline Efficiency in Cloud-based Big Data Systems. Journal of Big Data Analytics, 7(4), 367-384. https://doi.org/10.1007/s41060-022-00329-9

Mehta, P., Sharma, D., & Kumar, V. (2022). Serverless Data Processing with AWS Glue: An Analysis of Efficiency and Scalability. Journal of Cloud Services, 5(2), 112-126. https://doi.org/10.1016/j.jcs.2022.100372

Mohanty, S., Patra, B., & Bandyopadhyay, S. (2022). Advances in Cloud-based Data Pipeline Technologies: A Comprehensive Review. Big Data Research, 27(1), 1-14. https://doi.org/10.1016/j.bdr.2022.100233

Patel, A., & Joshi, M. (2022). Comparative Analysis of Data Integration Techniques Using Azure Data Factory. Journal of Data Management, 15(3), 367-389. https://doi.org/10.1016/j.jdm.2022.102456

Roy, P., Sinha, R., & Gupta, T. (2021). Cost Efficiency of AWS Glue in Large-Scale Data Projects. Journal of Cloud Economics, 8(2), 301-315. https://doi.org/10.1186/s13677-021-00290-y

Singh, A., & Kumar, S. (2020). Data Pipeline Optimization in Cloud Environments. Journal of Cloud Engineering, 6(3), 159-174. https://doi.org/10.1016/j.jce.2020.100267

Singh, S., Tiwari, R., & Gupta, M. (2023). AWS vs. Azure: Comparative Study for Data Pipeline Efficiency. IEEE Cloud Computing Magazine, 11(1), 15-28. https://doi.org/10.1109/MCC.2023.3264528

Wu, H., Li, X., & Wang, J. (2021). Cloud Computing and Big Data: A Review of the Current Trends. Journal of Cloud Computing Research, 8(2), 201-215. https://doi.org/10.1186/s13677-021-00244-y

Wu, Y., Zhang, T., & Li, P. (2021). A Comparative Analysis of AWS Glue and Azure Data Factory in Big Data Processing. Journal of Cloud Computing Research, 9(4), 201-215. https://doi.org/10.1186/s13677-021-00244-y

Yousuf, A., & Wei, Z. (2021). Tools and Techniques for Data Pipeline Optimization in Cloud Computing. Journal of Cloud Computing Research, 9(4), 215-230. https://doi.org/10.1186/s13677-021-00276-w

Zhang, L., & Li, Y. (2022). Comparing AWS and Azure for Big Data Processing Efficiency. International Journal of Cloud Engineering, 16(2), 67-83. https://doi.org/10.1007/s41060-022-00168-z

Downloads

Views

48

Downloads

2

Keywords:

Big Data Data Pipeline Efficiency AWS Glue Azure Data Factory Cloud Computing