Distributed Time Series Database Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently storing and retrieving large volumes of time series data, particularly in distributed environments, as raw storage lacks structure for easy retrieval and brute force searches become slow and resource-intensive when dealing with extensive data sets from multiple sources.
Innovation Solution
A distributed time series database system with a query layer, index database, and scalable architecture that uses a control node and data nodes to store and process data blocks, allowing for rapid storage and efficient retrieval by using metadata indices and sub-queries to identify and process relevant data blocks across multiple computing devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If raw storage is used for time series data, then storage capacity is sufficient, but retrieval efficiency deteriorates due to lack of structure
Solution Approach 1:
The patent segments the time series data into discrete data blocks organized in a hierarchical structure. Each data block contains a specific time range and is associated with metadata indices, allowing the system to divide large datasets into manageable, searchable units rather than storing all data as a single unstructured mass.
Solution Approach 2:
The patent introduces metadata indices as an intermediary layer between the raw time series data and the query interface. These indices contain aggregated statistical information (min, max, average values) that enable the system to quickly determine whether data blocks contain relevant results without scanning the actual time series data, thus mediating between storage and retrieval needs.
2Ease of operation
If brute force search methods are used, then all data can be searched, but resource consumption increases and performance deteriorates
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing metadata indices for each data block during the data ingestion phase. These indices contain aggregated statistical information that is prepared in advance, allowing the query system to quickly filter relevant data blocks without performing expensive computations during query execution.
Solution Approach 2:
The patent extracts essential statistical characteristics (min, max, average values) from the raw time series data and stores them separately as metadata indices. This extraction allows the system to work with compact summary information during queries rather than processing the entire raw dataset, significantly reducing resource consumption.
3Loss of information
If data from multiple sources is stored, then data completeness improves, but data volume grows quickly requiring larger storage
Solution Approach 1:
The patent segments data from multiple sources into distinct data blocks, each associated with specific metadata indices. This segmentation allows the system to organize heterogeneous data from multiple sources in a structured manner, maintaining data completeness while enabling efficient management and retrieval without requiring proportional increases in storage infrastructure.
4Adaptability or versatility
If distributed architecture is implemented, then scalability improves, but system complexity increases
Solution Approach 1:
The patent segments the distributed system into standardized components: data nodes for storage, query nodes for processing, and a coordinate system for organization. Each node operates independently with a clear interface, allowing the system to scale by simply adding more nodes following the same pattern, which manages complexity through standardization.
Solution Approach 2:
The patent designs data blocks and metadata indices as universal structures that can be replicated across multiple distributed nodes. The same data block format and index structure are used throughout the system, allowing any node to handle any data block, which simplifies the distributed architecture by eliminating the need for specialized node types.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system for storing time series data includes an ingester that prepares metadata indices associated with blocks of incoming time series data and stores the blocks of data in a time series database and the indices in a separate index database. The time series database distributes storage of the data blocks among multiple data nodes. A query layer receives queries and uses the index database to determine which data blocks are needed to process the query, and then requests only those data blocks from the time series database. Processing of the query is performed within the time series database only on those data nodes that contain relevant data, and partial results are passed to an output layer for formation into a final query result.