Data Engineering on Unstructured Dataset using AWS

US based OEM producing HVAC equipment, water heaters and boilers for residential and commercial buildings.

• The client has 120K+ live devices that send 40GB data per day in an unstructured format and it keeps on growing
• They wanted a scalable, secure and cost-effective solution with flexible architecture and intelligence to analyze such a large data

• Secure, scalable and flexible architecture having Serverless compute and storage
• Data collection, Extract-Transform-Load (ETL) and Data pipeline
• Data catalog and Database management
• Project based implementation with Infrastructure as Code (IaC)
• CloudFormation script for Infrastructure management
• CICD pipeline using GitHub Action
• Data collection script to transfer MongoDb data to S3
• Developed ETL script to convert unstructured data to structured format
• Dashboard development to display and monitor field devices data
• SSO authentication
• Analyze and visualize data

• Enabled faster data transformation by dividing day execution to hour execution for time series data
• Designed pipelines to process data for 1 year that continuously delivered meaningful insights to client