Reconstructing High-Resolution IoT Data Streams from Disparate Sources

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The proliferation of IoT devices generates large volumes of data streams that are often downsampled or deleted, limiting the quality of forensic analysis and machine learning retraining, as data is scattered across various locations and not all data is stored in high resolution, especially due to storage constraints and uncoordinated backups.

Innovation Solution

The system reconstructs high-resolution data streams by stitching together data from multiple sources using metadata like unique IDs or timestamps, allowing for the construction of desired data sizes and ensuring the reconstructed stream fits within database constraints, even if portions are low resolution or stored in disparate locations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is stored in time series database with down sampling policies, then storage capacity is improved, but data quality for forensics deteriorates

Engineering Contradiction:

Improvestorage capacityVSAvoiddata quality

Core Design Contradiction:

Quantity of substanceVSMeasurement precision

Solution Approach 1:

The system segments data storage into multiple locations: time series database for recent high-resolution data, data lakes for historical data, and cloud storage for archived data. This segmentation allows each storage location to maintain appropriate data quality for its purpose while optimizing overall storage capacity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different storage locations maintain different data qualities appropriate to their function. The time series database maintains high-resolution recent data for operational needs, while data lakes and cloud storage maintain lower-resolution historical data for forensic analysis and long-term retention

Inventive Principle:

Principle #3Local quality

2Productivity

If old data is deleted or down sampled, then storage efficiency is improved, but forensic analysis capability deteriorates

Engineering Contradiction:

Improvestorage efficiencyVSAvoidforensic data availability

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The system performs preliminary actions by continuously ingesting and storing data in data lakes and cloud storage before it is needed for forensic analysis. This ensures data is preserved in advance for potential future forensic requirements without impacting current storage efficiency

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Data lakes serve as an intermediary storage layer between the time series database and cloud storage. This intermediary preserves historical data in a cost-effective manner while maintaining the capability to retrieve it for forensic analysis when needed

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If data is scattered across multiple locations, then storage flexibility is improved, but data reconstruction complexity deteriorates

Engineering Contradiction:

Improvestorage flexibilityVSAvoidreconstruction complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system uses feedback mechanisms through metadata that tracks data location, timestamp, and resolution information. This feedback enables automated data reconstruction by providing the information needed to locate and retrieve appropriate data from multiple storage locations without manual intervention

Inventive Principle:

Principle #23Feedback

4Measurement precision

If high resolution data is retained, then forensic quality is improved, but storage requirements deteriorate

Engineering Contradiction:

Improveforensic data resolutionVSAvoidstorage requirements

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The system segments data retention by time and resolution, keeping only recent data in high-resolution format in the time series database, while storing older data in lower-resolution formats in data lakes and cloud storage. This segmentation maintains forensic capability for recent incidents while reducing overall storage requirements

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11681725B2Tiered forensics of IoT systems in cloud and time series databases

Publication Date: 2023.06.20 EMC IP HLDG CO LLC

AI summary

One example method includes creating an empty reconstruction stream database, identifying a data time interval, identifying data sources in which data was stored during the data time interval, reading data from the data sources, where the data read out from the data sources are associated with respective timestamps that fall within the data time interval, inserting the read out data into the empty reconstruction stream database so as to create a high resolution data stream, where the data are ordered in the empty reconstruction stream database according to timestamp, processing the data in the high resolution data stream and, based on the processing of the data, identifying and resolving a problem relating to an operating environment in which the data was initially generated.