Domain-Tailored Database for Real-Time IoT Sensor Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems for IoT sensor data struggle to efficiently detect domain-specific outliers, patterns, and events in real-time data streams, often requiring manual preprocessing and leading to biased analysis due to the dynamic nature of sensor data and large dataset sizes.
Innovation Solution
A domain-tailored database system that learns to identify outliers, patterns, and events using user-defined criteria, annotating data in real-time with bitmap indexes, allowing for declarative querying and automatic updating of definitions, enabling faster and more accurate data analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual preprocessing is used to detect outliers, patterns, and events in sensor data, then detection accuracy can be customized to domain requirements, but processing time increases and automation decreases
Solution Approach 1:
The system performs preliminary action by pre-computing bitmap indexes for outlier detection, pattern matching, and event detection during data ingestion. These indexes are built in advance using domain-specific criteria, allowing queries to retrieve results immediately without performing complex analysis at query time. This resolves the contradiction by shifting computation from query time to ingestion time, maintaining high detection accuracy while enabling fast query responses.
2Measurement precision
If complex domain-specific analysis is performed on large sensor datasets, then detection accuracy improves, but system complexity and computational resources increase
Solution Approach 1:
The system creates simplified copies of the data in the form of bitmap indexes that capture essential characteristics (outlier status, pattern presence, event occurrence) without storing the full complex analysis logic. These bitmap copies enable fast querying of domain-specific features without requiring the original complex computational resources, thus maintaining detection accuracy while reducing system complexity during query operations.
3Productivity
If real-time detection of outliers, patterns, and events is implemented, then productivity improves, but measurement precision may decrease due to dynamic data changes
Solution Approach 1:
The system implements dynamics by allowing the redefinition of outlier criteria, patterns, and events without requiring system restructuring. Users can update domain-specific definitions and the system adapts by recomputing bitmap indexes with new criteria. This enables real-time detection productivity while maintaining measurement precision through flexible, updatable domain-specific logic that adapts to changing data characteristics.
4Speed
If comprehensive bitmap indexes are created for all detection criteria, then query speed improves, but memory usage and storage requirements increase
Solution Approach 1:
The system extracts only the essential detection results into compact bitmap indexes rather than storing complete analysis data. Each bitmap index captures binary outcomes (outlier/not outlier, pattern present/absent, event occurred/not occurred) for each data point. This extraction approach enables fast query speed by working with compact binary representations while significantly reducing storage requirements compared to storing full analysis results or raw data with annotations.
Data Source
AI summary
Methods and apparatus are provided for domain-tailored detection of outliers, patterns, and/or events in data streams. An exemplary method comprises obtaining a domain-dependent definition of (i) data outliers based on predefined outlier criteria; (ii) data patterns based on predefined pattern criteria; and/or (iii) data events based on predefined event criteria; obtaining time series measurement data from a plurality of sensors; determining, substantially simultaneously with the obtaining, whether individual samples satisfy the domain-dependent definitions of the data outliers, data patterns and/or data events; and storing the individual samples with an indication of whether the individual samples satisfy the domain-dependent definitions of the data outliers, data patterns and/or data events. The domain-dependent definitions are optionally specified using a declarative command language. Query are optionally processed comprising one or more declarative statements that reference and/or manipulate the data outliers, data patterns and/or data events.