Time Series Data Capture with Heterogeneous Sensor Profiles

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current systems for capturing and storing time series data from remote sensors with heterogeneous reporting profiles face challenges such as irregular data arrival, high data influx, lack of long-term storage, and inadequate query and transformation capabilities, limiting their ability to handle complex data analysis and retrieval effectively.

Innovation Solution

A distributed system with a data stream management engine and a multidimensional time series data store that aggregates and stores sensor data using a key-value pair format, allowing for transparent data transfer across multiple access connections, long-term archiving, and robust SQL-like query language for complex data retrieval and transformation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is stored in a straight key-value pair data store with fixed retention period, then storage capacity is maintained at manageable levels, but long-term storage capability is lost

Engineering Contradiction:

Improvedata storage capacityVSAvoiddata retention period

Core Design Contradiction:

Quantity of substanceVSDuration of action of stationary object

Solution Approach 1:

The system divides the data store into two distinct segments: a short-term data store (straight key-value pair) for recent data and a long-term data store for archived data. This segmentation allows each segment to be optimized for its specific purpose while collectively providing both manageable storage capacity and extended data retention.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

An archival management component acts as an intermediary between the short-term and long-term data stores. This mediator automatically transfers data between stores based on retention policies, maintaining the benefits of both short-term quick access and long-term preservation without requiring manual intervention.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If data aggregation blocks include readings from arbitrary subpopulation of sensors, then storage efficiency is improved, but data completeness for specific sensor queries is reduced

Engineering Contradiction:

Improvestorage efficiencyVSAvoidsensor data completeness

Core Design Contradiction:

Quantity of substanceVSLoss of information

Solution Approach 1:

The query language incorporates feedback mechanisms that allow users to specify exact sensor identifiers and time ranges. The system uses this feedback to precisely locate and retrieve data from the aggregated blocks, ensuring complete sensor data retrieval even when data is stored in aggregated form across multiple blocks.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system adds temporal and sensor-identifier dimensions to the aggregated data blocks. By organizing data with multiple indexing dimensions (time, sensor ID, data type), the system enables efficient retrieval of specific sensor data from aggregated storage without sacrificing storage efficiency.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Ease of operation

If strictly time interval based commitment schedule is used, then data capture is simple and reliable, but handling of irregular sensor data arrivals becomes problematic

Engineering Contradiction:

Improvedata capture simplicityVSAvoidhandling of heterogeneous reporting profiles

Core Design Contradiction:

Ease of operationVSAdaptability or versatility

Solution Approach 1:

The commitment schedule is made dynamic by allowing multiple aggregation strategies (time-based, count-based, size-based) to be selected based on sensor characteristics. The system automatically adapts the aggregation parameters for each sensor or sensor group, maintaining operational simplicity while accommodating heterogeneous reporting profiles.

Inventive Principle:

Principle #15Dynamics

4Device complexity

If no transformative capabilities are provided, then system complexity is reduced, but ability to analyze noisy and complex data is limited

Engineering Contradiction:

Improvesystem complexityVSAvoiddata analytical value

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The query language is designed to be universal, combining data retrieval, filtering, aggregation, and transformation capabilities in a single standardized interface. This multi-functional query language provides comprehensive data analysis capabilities without requiring separate complex processing systems, thus managing overall system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10204147B2System for capture, analysis and storage of time series data from sensors with heterogeneous report interval profiles

Publication Date: 2019.02.12 QOMPLX INC

AI summary

A system and method for the capture, storage and analysis of multidimensional time series data from sources with heterogeneous reporting profiles has been devised. Data from sets of sensors that send variable amounts of multidimensional data at irregular intervals is received by a data processing device which processes the raw data to extract the parameters of interest and held until a preset number of sensor events or amount of stream data is received. The data is then committed to a data store until time of analysis. Older data is written to archive storage. The system allows complex selection and transformation of data store data by robust query language.