Abstract
In cloud computing, data is generated from various sources in different locations, such as database management systems, data streaming environments, and files, and is automatically transferred to a data lake. Managing that data in the data lake is a challenging problem that involves developing a scalable model to integrate, process, and move it from one source to another with optimized cost and performance. The earlier research used various optimisation, data management and processing tools, but their performance efficiency is poor. This paper has been motivated to improve the overall performance by implementing the Random Forest algorithm for an overview of the incoming data, which is passed into a cost-effective data pipeline that helps to ingest and process massive amounts of incoming data daily. It also involves Apache Spark cloud services integrating the data seamlessly to manage storage and processing. Based on the data privacy and governance policies the RF model monitors and secures sensitive data in the input dataset. The simulation output is obtained by executing the proposed model in the cloud Apache framework, and the efficiency is verified regarding execution time, response time for user query, and accuracy. It is also verified that the data transmission with the cost efficiency for the healthcare dataset simulated in the Spark, S3, and AWS lake provides an aware pipeline model. A large-scale healthcare dataset is used in the simulation to confirm the efficacy of the data pipelining model, and the data transmission rate, claims, and cost efficiency are verified.