Time Series Data Storage Indexing for Query Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid generation of large volumes of time series data from industrial equipment and systems poses challenges in storage and retrieval, as raw storage lacks structure for easy retrieval, and brute force searches become slow and resource-intensive when dealing with extensive data sets.
Innovation Solution
A system comprising an ingester, index database, time series database, query layer, evaluator, and output handler is implemented, which processes and stores time-stamped data across multiple computing devices, allowing for efficient storage and retrieval by creating indices and sub-queries to minimize data movement and processing overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If raw storage is used for time series data, then storage capacity is sufficient, but retrieval efficiency deteriorates due to lack of structure
Solution Approach 1:
The patent segments the time series data storage system into multiple components: raw data storage, index structures, and query processing layers. Data is divided into time-based partitions and tagged with metadata, allowing structured organization without sacrificing storage capacity. This segmentation enables efficient retrieval by allowing queries to target specific time ranges and tags rather than scanning all data.
Solution Approach 2:
The patent introduces index structures as intermediary elements between the raw storage and query interfaces. These indexes contain metadata about the time series data (time ranges, tags, measurement types) and enable the query system to quickly locate relevant data without scanning the entire storage volume, thus improving retrieval speed while maintaining full storage capacity.
2Adaptability or versatility
If brute force search methods are used, then all data can be searched, but resource consumption increases and performance deteriorates
Solution Approach 1:
The patent implements preliminary action by pre-computing and storing index structures that organize data by time ranges, tags, and measurement types before queries are executed. This preliminary organization allows the query system to quickly identify and access only the relevant data subsets matching query criteria, avoiding the need to scan all data and significantly reducing resource consumption while maintaining comprehensive search coverage.
3Volume of stationary object
If data is stored across multiple computing devices, then storage scalability improves, but system complexity increases
Solution Approach 1:
The patent implements a unified query interface that works consistently across distributed storage. The index structures and query processing logic provide universal access patterns that abstract away the underlying distribution complexity, allowing the system to scale storage across multiple devices while maintaining a simple, consistent interface for data retrieval operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A service for storing time series data provides a data pipe for receiving time series data, a query pipe for making requests to the service, and a result pipe for receiving output from the service. Data sent to the query pipe is processed by an ingester that prepares metadata indices associated with blocks of incoming time series data and stores the blocks of data in a time series database and the indices in a separate index database. A query layer receives queries from the query pipe and uses the index database to determine which data blocks are needed to process the query, and then requests only those data blocks from the time series database. Processing of the query is performed within the time series database only on those data nodes that contain relevant data, and partial results are passed to an output layer for formation into a final query result which is sent out by the results pipe.