Multi-Representation Time Series Storage Advisor
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume and frequency of time-series data from industrial processes and IoT devices pose challenges in efficient storage and analysis, particularly in managing memory consumption and query response times, as existing approaches often conflict between execution speed and memory requirements.
Innovation Solution
A multi-representation storage approach for time-series data that selectively applies different compression models, error bounds, and storage media along the time axis (horizontal) and across the same period (vertical), optimized by an advisor that considers user configurations, workload patterns, and cost models to create 'hot zones' for efficient access and data aging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single compression model is used for all time series data, then storage efficiency is improved, but query response time deteriorates
Solution Approach 1:
The patent segments time series data into multiple representations based on time periods (horizontal segmentation) and data characteristics (vertical segmentation). Different compression models are applied to different segments, allowing efficient storage for historical data while maintaining fast access for recent data through appropriate representation selection.
Solution Approach 2:
The patent applies different compression models and representations to different portions of time series data based on local requirements. Recent data or frequently queried data receives representations optimized for fast access, while historical data uses representations optimized for compression, achieving local optimization rather than uniform treatment.
2Speed
If multiple representations are stored for the same time period, then query response time is improved, but memory consumption increases
Solution Approach 1:
The patent dynamically selects which representations to store and maintain based on current query workloads and data access patterns. The system adapts the number and type of representations stored, converting between representations as needed, rather than statically maintaining all possible representations simultaneously.
Solution Approach 2:
The patent creates copies of time series data in different representations as needed for specific query types. Rather than maintaining all representations permanently, the system generates appropriate representations on-demand or maintains only the most frequently needed ones, reducing overall memory consumption.
3Measurement precision
If high accuracy is maintained for all data, then data quality is improved, but storage requirements increase
Solution Approach 1:
The patent applies different accuracy levels periodically or based on data age, maintaining high accuracy for recent data while using compressed representations with lower accuracy for historical data. This periodic differentiation in quality requirements reduces overall storage needs while preserving data quality where most needed.
Data Source
AI summary
An advisor creates configurations for generating multi-representations of time series data based upon detected characteristics such as length, interval, minimums, data types, etc., as well as configurations provided by a user. In an offline mode the advisor may further consider a previous time series workload. In an on-line mode the advisor may adapt multi-representation configurations with respect to ongoing changes in a current time series workload. The advisor may reference a cost model including values quantifying various dimensions (e.g., compression technique, accuracy, covered time period, storage medium, memory consumption, speed) of the multi-representations for optimization purposes. Configurations created by the advisor may be input to a storage engine to generate and store the multi-representations according to goals for data aging, operation execution pattern optimization, and ease of access to time series data located in hot zones. The advisor may be implemented with an engine of an in-memory database.