IoT Data Ingestion System with Unified Format Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current IoT management systems fail to seamlessly integrate with IoT devices, struggle to process large volumes of unstructured data, and require significant time and resources to update, leading to potential hazardous conditions and legal liabilities in industries like the chemical industry.
Innovation Solution
A data ingestion system that receives and converts telemetry data from IoT devices into unified formats, identifies anomalies in near-real time, and performs actions based on stored data, using cloud-based serverless components to process data efficiently and adapt to changing business needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current IoT management systems are used to integrate and process IoT device data, then system integration is achieved, but the systems fail to seamlessly integrate with diverse IoT devices and struggle to process large volumes of unstructured data efficiently
Solution Approach 1:
The system segments data processing into multiple specialized components: a data ingestion service for receiving telemetry data, a data normalization service for converting to unified formats, and an anomaly detection service for real-time analysis. This segmentation allows each component to optimize for its specific function, improving overall processing efficiency while maintaining adaptability to diverse device formats.
Solution Approach 2:
The patent introduces a data normalization service as an intermediary layer between diverse IoT devices and the processing system. This intermediary converts various device-specific data formats into unified formats, enabling seamless integration with diverse devices without requiring changes to the core processing infrastructure, thus maintaining both adaptability and efficiency.
2Reliability
If traditional IoT management systems are used, then basic data collection is possible, but significant time and resources are required to update and maintain the systems
Solution Approach 1:
The system implements self-service capabilities through automated anomaly detection that continuously monitors telemetry data and identifies issues without human intervention. The serverless architecture automatically scales resources based on demand, eliminating the need for manual system updates and maintenance, thus improving reliability while reducing time and resource investment.
Solution Approach 2:
The anomaly detection service provides continuous feedback by analyzing telemetry data in real-time and identifying potential issues before they become critical problems. This feedback mechanism enables proactive maintenance, reducing the need for reactive system updates and minimizing downtime, thereby improving reliability while reducing maintenance time and resources.
3Speed
If real-time anomaly detection is implemented, then faster decision-making is enabled, but computing and networking resources are consumed
Solution Approach 1:
The system uses serverless computing architecture that dynamically allocates computing resources based on actual demand. When telemetry data volume is low, resources are minimized; when data volume increases or anomalies are detected, resources automatically scale up. This dynamic resource allocation enables real-time anomaly detection speed while optimizing energy and computing resource consumption.
4Reliability
If large volumes of unstructured data are processed, then comprehensive monitoring is achieved, but processing efficiency decreases and resource consumption increases
Solution Approach 1:
The data normalization service extracts and converts only the essential telemetry data points from large volumes of unstructured data into unified formats. By extracting only the relevant information needed for anomaly detection and monitoring, the system achieves comprehensive monitoring coverage while maintaining high processing efficiency and reducing resource consumption associated with processing unnecessary data.
Data Source
AI summary
A device may receive, from Internet of Things (IOT) devices, telemetry data in a variety of formats and may convert the telemetry data to converted telemetry data in a first unified format. The device may receive, from one or more of the IoT devices, offline IoT data in a variety of formats and may convert the offline IoT data to converted offline IoT data in a second unified format. The device may receive events from one or more of the IoT devices and may identify anomalies based on the telemetry data and in near-real time relative to receiving the telemetry data. The device may store the converted telemetry data, the converted offline IoT data, the events, and the anomalies, as stored data, in a data structure, and may perform actions based on the stored data.