--- title: Serverless Stream Data Pipeline in AWS description: In this video, Andreia Negreira discusses the scope, technical details, and her insights from her AWS internship project at Big Industries. image: https://www.bigindustries.be/hubfs/Andreia%20Internship-1-1.png --- ###### [Big Industries Academy](https://www.bigindustries.be/blog/tag/big-industries-academy) # Serverless Stream Data Pipeline in AWS ![Andreia Negreira](https://www.bigindustries.be/hs-fs/hubfs/Andreia%20Negreira-1.jpeg?width=56&name=Andreia%20Negreira-1.jpeg) [ Andreia Negreira ](https://www.bigindustries.be/blog/author/andreia-negreira) Aug 13, 2024 4:34:29 PM SHARE [mailto:?subject=Check%20out%20Serverless%20Stream%20Data%20Pipeline%20in%20AWS%20&body=Check%20out%20https://www.bigindustries.be/blog/serverless-stream-data-pipeline-in-aws](mailto:?subject=Check%20out%20Serverless%20Stream%20Data%20Pipeline%20in%20AWS%20&body=Check%20out%20https://www.bigindustries.be/blog/serverless-stream-data-pipeline-in-aws)   In this video, Andreia Negreira discusses the scope, technical details, and her insights from her AWS internship project at Big Industries. The project leverages Amazon MSK for data streaming with AWS Lambda acting as both producers and consumers. It ingests data via the OpenSky API, orchestrates it using various AWS services, and persists it to Amazon S3 and RDS, employing a range of AWS offerings to manage data securely and effectively. #### Architecture ![Serverless Stream Data Pipeline Architecture-1](https://www.bigindustries.be/hs-fs/hubfs/Serverless%20Stream%20Data%20Pipeline%20Architecture-1.png?width=724&height=358&name=Serverless%20Stream%20Data%20Pipeline%20Architecture-1.png) *Figure 1: Project Architecture* - **Data Ingestion**: Data is ingested from the OpenSky API via AWS API Gateway. - **Data Processing**: An AWS Lambda function acts as the producer, sending data to Amazon MSK (Managed Streaming for Apache Kafka). - **Data Storage**: Data is stored in Amazon S3 and Amazon RDS by the consumer Lambdas. - **Security**: The main VPC ensures secure communication, with an Internet Gateway providing necessary external access. An Amazon EC2 bastion host allows secure access to the MSK cluster. #### Detailed Architecture ![Detailled Architecture-1](https://www.bigindustries.be/hs-fs/hubfs/Detailled%20Architecture-1.png?width=1000&height=486&name=Detailled%20Architecture-1.png) *Figure 2: Project Architecture Details* - **Cluster Authentication**: SASL/SCRAM (Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism) is used for authentication, managed by AWS Secrets Manager. - **Secure Communication**: TLS certificates and SCRAM profiles are managed within the cluster to ensure secure communication. #### Technologies Used - **AWS API Gateway:** To manage the OpenSky API endpoints. - **AWS Lambda**: For serverless computing, used as both producer and consumer. - **Amazon MSK**: Managed Kafka service for stream processing. - **Amazon S3**: For scalable object storage. - **Amazon RDS**: For relational database storage. - **AWS VPC**: For creating a secure network environment. - **AWS EC2**: For the bastion host to manage cluster security. - **AWS Secrets Manager**: For managing SASL/SCRAM credentials. - **Terraform**: For infrastructure as code, automating the setup of AWS resources. - **OpenSSL**: For generating TLS certificates. #### Contact For any questions or issues, contact Andreia Negreira at [andreia.negreira@bigindustries.be](mailto:andreia.negreira@bigindustries.be). [![Need help with your Data & Analytics Project?](https://hubspot-no-cache-eu1-prod.s3.amazonaws.com/cta/default/2240994/a4582ae2-d182-4146-96c1-24f066a33fca.png)](https://hubspot-cta-redirect-eu1-prod.s3.amazonaws.com/cta/redirect/2240994/a4582ae2-d182-4146-96c1-24f066a33fca)   ![Andreia Negreira](https://www.bigindustries.be/hs-fs/hubfs/Andreia%20Negreira-1.jpeg?width=66&name=Andreia%20Negreira-1.jpeg) #### [ Andreia Negreira ](https://www.bigindustries.be/blog/author/andreia-negreira) In June 2023 Andreia joined the AI class at BeCode, where she found the perfect environment for a hands-on training, enhancing her problem-solving skills cultivated during her studies in Environmental Engineering and her master's in chemical and Biochemical Process Engineering. With experience in data integration, streaming, ETL design, and end-to-end data science pipelines, she is able to bring a fresh perspective and a constant passion for innovation to the tech world, fueled by her analytical and problem-solving skills developed throughout her career. ## Related posts At Big Industries, we want to share our wisdom. More info on our recent developments, newest projects and upcoming events [![RabbitMQ Logo](https://www.bigindustries.be/hubfs/RabbitMQ-1.png) ](https://www.bigindustries.be/blog/exploring-messaging-and-streaming-technologies-part8-rabbitmq) ###### [Big Industries Academy](https://www.bigindustries.be/blog/tag/big-industries-academy) #### [Exploring Messaging and Streaming Technologies Part8: RabbitMQ](https://www.bigindustries.be/blog/exploring-messaging-and-streaming-technologies-part8-rabbitmq) As Francine Anestis continues her deep dive into Messaging and Streaming technologies, she delves... [Read more ](https://www.bigindustries.be/blog/exploring-messaging-and-streaming-technologies-part8-rabbitmq) [![Google Cloud Pub/Sub](https://www.bigindustries.be/hubfs/Google%20Cloud%20PubSub-1.png) ](https://www.bigindustries.be/blog/exploring-messaging-and-streaming-technologies-part5-google-pub/sub) ###### [Big Industries Academy](https://www.bigindustries.be/blog/tag/big-industries-academy) #### [Exploring Messaging and Streaming Technologies Part5: Google Pub/Sub](https://www.bigindustries.be/blog/exploring-messaging-and-streaming-technologies-part5-google-pub/sub) This is the fifth article in a series of articles by Francine Anestis, where she looks at different... [Read more ](https://www.bigindustries.be/blog/exploring-messaging-and-streaming-technologies-part5-google-pub/sub) #### Subscribe to our newsletter Stay informed about our recent developments, newest projects and upcoming events ![img-mails](https://f.hubspotusercontent10.net/hubfs/369261/raw_assets/public/Marketplace/leadstreet/themes/popular-theme/images/img-mails.svg) ## Ready to set off on a BIG journey? ##### The top notch technologies we use set us apart from other consultancies [HAVE A LOOK AT OUR SERVICES](https://www.bigindustries.be/services) ```json { "@context" : "https://schema.org", "@type" : "VideoObject", "caption" : { "@type" : "MediaObject", "contentUrl" : "https://www.bigindustries.be/media-transcripts/175669996141/en.vtt", "inLanguage" : "en", "name" : "en Captions" }, "contentUrl" : "https://2240994.fs1.hubspotusercontent-eu1.net/hubfs/2240994/technical_kafka_project_video.mp4", "dateModified" : "2025-08-05T15:18:23.146Z", "duration" : "PT5M56S", "height" : 1080, "name" : "technical_kafka_project_video", "thumbnailUrl" : "https://2240994.fs1.hubspotusercontent-na1.net/hubfs/2240994/technical_kafka_project_video.mp4/medium.jpg?t=1754407103146", "uploadDate" : "2024-08-14T07:46:04.620Z", "width" : 1920 } ```