Sample use case: Collect sensor data from the IOT asset (like turbine) & Build insights on short-term optimization as well as longer term maintenance costs -by ingesting data into the cloud for storage and analytics.
IoT Architecture Main Components: (please map below sequence number with numbers in diagram )
- Azure IoT Hub (recommended) or Event Hub– : For time-series streaming sensor data ingestion from IoT devices into Azure.
- Azure Data Factory: For ingestion and data orchestration of structured maintenance and failure data collected in a batch process.
- Azure Data Lake Store (ADLS): stores raw data (‘write-once, access-often’ pattern) & databricks uses delta storage format for a layer having ACID transaction, resiliency, and performance on the ADLS.
- Azure Databricks (ADB): Uses Apache Spark & Delta architecture for big data analytics for data engineering and data science. Using delta architecture, it unifies the streaming & batch data by creating Bronze/Silver/Gold layer
- Bronze delta table: store raw IoT/batch data
- Silver delta table: store refined/clean/combined with look up data commonly used for machine learning and data science. Feature engineering & feature selection done here to ready datasets to be consumed by Azure ML
- Gold delta table: store aggregated/enriched the data ready for analytics and reporting purposes
- Azure Machine Learning: dataset created in 4B loaded in Azure Machine Learning to build a predictive maintenance model or a power generation prediction mode
- Azure Synapse Analytics: data from gold table loaded into Azure Synapse Analytics for BI analytics and reporting along with Power BI
- Power BI: to visualize the data stored in Synapse Analytics.