Sensor Data Annotation Schema for IoT Analytics Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in effectively processing and analyzing data from Internet of Things (IOT) devices, which often lack descriptive information and are in semi-structured formats, making it difficult to combine with structured data for analytical purposes, especially in cloud-based systems with diverse data sources and schemas.
Innovation Solution
The solution involves annotating IOT data with metadata elements, using a relational database schema to store and process this data, and implementing a pipeline that efficiently handles IOT data from multiple sources, allowing for aggregation, storage, and analysis while maintaining data separation and optimizing for analytics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If IOT data is stored in semi-structured formats without metadata annotation, then storage and initial ingestion are simpler, but data processing and analysis become difficult and inefficient
Solution Approach 1:
The patent applies preliminary action by annotating IOT data with metadata elements during the initial ingestion phase, before processing and analysis operations. This pre-annotation includes adding device identifiers, sensor types, units, and other contextual information, so that subsequent data processing operations can efficiently utilize this structured information without requiring complex real-time processing or transformation operations.
2Adaptability or versatility
If diverse IOT data sources with different schemas are integrated, then data comprehensiveness improves, but system complexity and processing difficulty increase
Solution Approach 1:
The patent applies parameter changes by standardizing diverse IOT data schemas through a common metadata annotation framework. Different data sources with varying schemas are transformed into a unified format by applying consistent metadata elements (device_id, sensor_type, unit, timestamp, etc.), allowing the system to handle diverse inputs without requiring complex source-specific processing logic for each data type.
Solution Approach 2:
The patent uses an intermediary approach by introducing a standardized metadata annotation layer between diverse IOT data sources and the processing system. This intermediary layer translates various data schemas into a common format, acting as a mediator that enables seamless integration of heterogeneous data sources while shielding the core processing system from schema diversity complexity.
3Loss of information
If detailed metadata is added to IOT data, then data context and analyzability improve, but data volume and processing overhead increase
Solution Approach 1:
The patent applies the extraction principle by separating metadata elements from the core sensor data while maintaining their association. Critical contextual information (device_id, sensor_type, unit) is extracted and stored as structured metadata annotations, allowing the system to preserve data context without duplicating full data copies. This separation enables efficient storage and processing by treating metadata as reference information rather than redundant data.
Data Source
AI summary
Techniques for processing sensor data are provided. Sensor data, such as individual messages or data points from devices having one or more hardware sensors, can be annotated with one or more metadata elements to facilitate sensor data processing. An annotation rule for sensor data can be determined and sensor data annotated according to the annotation rule. Sensor data can be written to a relational database table, where the table has a schema that provides columns for storing data for particular indicators of an indicator group having a plurality of indicators.