Database Fragment Consolidation for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database management systems face challenges in efficiently managing large data sets due to increasing data capacity requirements, where existing hybrid approaches for in-memory and on-disk systems often lead to blocking situations and inefficient query processing.
Innovation Solution
The method involves storing data records in main part fragments, where old fragments are loaded into memory for deletion and merging, and adjacent fragments are joined when small, using a unified architecture that optimizes compression and indexing, allowing for on-demand paging and multi-segmented indexing to improve performance and reduce storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If old fragments are kept on-disk to reduce storage costs, then storage efficiency improves, but query performance deteriorates due to frequent disk accesses
Solution Approach 1:
The main part is divided into multiple fragments that are further divided into hot and cold segments. Hot fragments are kept in memory for fast access, while cold fragments are stored on-disk. This segmentation allows the system to maintain both high query performance for frequently accessed data and cost-effective storage for less frequently accessed data.
Solution Approach 2:
The system dynamically manages fragment locations by moving fragments between memory and disk based on access patterns and fragmentation thresholds. This dynamic approach allows the system to adapt to changing data access requirements and maintain optimal performance while managing storage resources efficiently.
2Reliability
If merge process is performed frequently to remove deleted records, then data integrity improves, but system downtime increases
Solution Approach 1:
The system performs preliminary actions by marking records for deletion rather than immediately removing them. Deleted records are tracked in a log and removed during scheduled merge operations. This preliminary marking approach allows the system to maintain data integrity without requiring immediate merge operations, thereby reducing system downtime.
Solution Approach 2:
The system continues to accept new deletions and updates even when merge operations are in progress or scheduled. The delta part accumulates changes continuously, and merge operations proceed in the background or during maintenance windows. This continuous operation approach maintains data integrity while minimizing disruption to ongoing database operations.
3Quantity of substance
If fragments are merged to remove deleted records, then storage space utilization improves, but merge process time increases
Solution Approach 1:
By segmenting the main part into multiple fragments with hot and cold divisions, the system reduces the size of individual merge operations. Only cold fragments need to be merged with the delta part, while hot fragments remain in memory and are updated more frequently. This segmentation significantly reduces merge process time while still achieving effective storage space utilization.
Solution Approach 2:
The system performs merge operations periodically rather than continuously. Merge operations are scheduled based on fragmentation thresholds and system load conditions. This periodic approach allows the system to accumulate deletions and updates in the delta part, then process them in batches, reducing the frequency and duration of merge operations while maintaining storage space utilization.
4Speed
If in-memory storage is used to improve query performance, then query performance improves, but storage cost increases
Solution Approach 1:
The system segments data into hot and cold fragments, keeping only the most frequently accessed data in expensive in-memory storage while storing less frequently accessed data on cheaper disk storage. This segmentation allows the system to optimize query performance for critical data while reducing overall storage costs by utilizing lower-cost storage for less critical data.
Solution Approach 2:
The system applies different storage qualities to different data segments. Hot fragments receive high-performance in-memory storage for fast access, while cold fragments use cost-effective on-disk storage. This local quality differentiation allows the system to allocate storage resources based on actual data access patterns, optimizing both performance and cost efficiency.
Data Source
AI summary
A plurality of data records of a data set can be stored in a plurality of main part fragments, at least one of which is an old fragment stored on-disk. A number of one or more data records in the old fragment that have been marked for deletion can be determined to be greater than a threshold number, and the old fragment can be loaded into main system memory. A merge of the old fragment can be performed to remove the one or more data records marked for deletion.