Time Series Database Data Hole Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Time series databases face challenges in data lifecycle management, particularly in efficiently removing and reusing data holes created by periodic deletion of unused or garbage data, which affects storage and performance.
Innovation Solution
A computer-implemented method and system that updates metadata and data hole indices to indicate soft removal of data, optimizes data holes based on input/output rates, and reuses data holes for storage and read/write benefits by determining whether data holes fit the needs for new data writes and moving frequently accessed data to these holes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is periodically deleted from time series database to manage data lifecycle, then storage efficiency is improved, but data holes are created that affect read/write performance
Solution Approach 1:
The patent implements soft deletion of data by updating metadata indices to mark data as deleted while preserving the physical storage space. These recovered storage spaces (data holes) are then reused for new data writes, eliminating the need to allocate new storage blocks and improving write performance while maintaining storage efficiency
Solution Approach 2:
The system performs preliminary actions by pre-allocating storage spaces and maintaining data hole indices that track available reuse locations. When data is marked for deletion, the storage space is immediately made available for future writes through the data hole index, preventing future allocation delays and improving overall read/write performance
2Quantity of substance
If data holes are optimized when I/O rate is low, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements periodic optimization of data holes triggered by I/O rate thresholds. When the I/O rate falls below a specified threshold, the system activates data hole optimization processes that consolidate and reorganize storage spaces. This periodic action balances storage efficiency improvements with system complexity by only performing optimization when system is under low load
Solution Approach 2:
The system changes operational parameters by adjusting the I/O rate threshold that triggers data hole optimization. This parameter-based control allows the system to dynamically adapt its complexity level based on current workload conditions, performing storage optimization only when I/O demand is sufficiently low to accommodate the additional processing
Data Source
AI summary
An indication to remove one or more data in a time series database is received. A metadata index associated with the time series database is updated to indicate a soft removal of each data of the one or more data. A data hole index associated with the time series database is updated to indicate a data hole at a location of each data of the one or more data in the time series database. Responsive to an input/output rate for the time series database being below a threshold, the data hole of each data of the one or more data is optimized.