Home-Automation

Data Engineering on Unstructured Dataset using AWS

Client Overview

US based OEM producing HVAC equipment, water heaters and boilers for residential and commercial buildings.

Business Need

  • The client has 120K+ live devices that send 40GB data per day in an unstructured format and it keeps on growing
  • They wanted a scalable, secure and cost-effective solution with flexible architecture and intelligence to analyze such a large data

VOLANSYS Contribution

  • Secure, scalable and flexible architecture having Serverless compute and storage
  • Data collection, Extract-Transform-Load (ETL) and Data pipeline
  • Data catalog and Database management
  • Project based implementation with Infrastructure as Code (IaC)
  • CloudFormation script for Infrastructure management
  • CICD pipeline using GitHub Action
  • Data collection script to transfer MongoDb data to S3
  • Developed ETL script to convert unstructured data to structured format
  • Dashboard development to display and monitor field devices data
  • SSO authentication
  • Analyze and visualize data

Solution Diagram

EXT-New-261

Benefits Delivered

  • Enabled faster data transformation by dividing day execution to hour execution for time series data
  • Designed pipelines to process data for 1 year that continuously delivered meaningful insights to client
Similar Success Stories