Time Series Database Two-Level Tree Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing time series database systems face inefficiencies in writing and querying data due to high memory costs, limited storage capacity, and poor performance in handling out-of-order data and stale data clearing, particularly with two-level index structures and binary sort trees.
Innovation Solution
A two-level tree structure is implemented where time series data are stored in leaf nodes, allowing for efficient querying with a time complexity of O(1) at the root node and O(log N) in leaf nodes, reducing memory usage, and enabling efficient clearing of stale data and out-of-order writing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If two-level index structure with string storage is used, then data can be organized for rapid reading, but write performance deteriorates due to capacity constraints requiring data copying
Solution Approach 1:
The patent divides time series data into multiple time windows (e.g., hourly, daily) and stores them in separate memory regions. Each time window is independently managed, allowing new data to be appended without copying existing data, thus improving write performance while maintaining efficient reading within each segment.
Solution Approach 2:
The patent implements a dynamic memory management strategy where memory regions are allocated and freed based on actual data volume and access patterns. This allows the system to adapt to varying data sizes and maintain optimal performance for both reading and writing operations.
2Speed
If all time series data are stored in memory for rapid access, then reading speed improves, but memory cost increases and storage capacity is exceeded
Solution Approach 1:
The patent segments time series data into different time windows (e.g., recent data in memory, historical data in disk storage). This allows the system to maintain only recently accessed data in memory for rapid reading, while storing older data on disk, thus reducing memory footprint while maintaining fast access for recent queries.
Solution Approach 2:
The patent implements periodic data migration strategies where data is moved between memory and disk storage based on time windows and access patterns. This ensures that frequently accessed recent data remains in memory while less frequently accessed historical data is moved to disk, optimizing the balance between reading speed and memory usage.
3Ease of operation
If binary sort tree is used for data organization, then data can be sorted by timestamps for convenient querying, but query performance deteriorates in worst case scenarios and memory consumption increases
Solution Approach 1:
The patent divides the time series data into multiple time windows and stores them in separate sorted lists or arrays. This segmentation allows for efficient binary search or binary partitioning within each time window, achieving O(log N) query performance while using less memory than a complete binary sort tree would require.
Solution Approach 2:
The patent changes the data organization parameter from a hierarchical binary sort tree to a flattened time window-based sorted structure. This parameter change maintains the ability to query by time range while improving query performance and reducing memory consumption through more efficient data layout and indexing.
4Device complexity
If index structure only stores time series ids is used, then structure is simple, but clearing stale data requires looping through all time series resulting in low efficiency
Solution Approach 1:
The patent segments the index structure into time window-based indexes where each time window maintains its own list of time series ids. This allows the system to clear stale data by processing only the relevant time windows instead of looping through all time series, significantly improving clearing efficiency while maintaining reasonable index structure complexity.
Solution Approach 2:
The patent implements preliminary organization of data into time windows with associated time series ids during the data ingestion phase. This preliminary structuring enables efficient stale data clearing by pre-grouping data temporal characteristics, avoiding the need to scan through the entire index during the clearing operation.
Data Source
AI summary
An objective of the present disclosure is to provide a method and apparatus for accessing time series data. The method according to the present disclosure comprises: acquiring time series query information; determining corresponding time offset information based on the time series query information, the time offset information including hour information, minute information, and offset information; finding a leaf node in a time tree corresponding to the time offset information based on the time offset information; and querying in the leaf node based on the offset information, to acquire corresponding time series data. Advantages of the present disclosure lie in possessing a lower complexity and improving query efficiency of the time series data. In addition, the memory is less occupied, and out-of-order writing is supported.