Database Compaction Policy for Distributed Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional compaction techniques in distributed databases often waste system resources by performing frequent or non-optimal compactions, leading to increased costs and latency in storage systems.
Innovation Solution
A compaction policy that selectively merges rowsets based on physical layout and cost analysis to optimize storage efficiency, minimizing I/O operations and maintaining consistent system performance by placing soft limits on file sizes and using incremental compaction strategies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compaction techniques are used frequently, then data storage efficiency is improved, but system resources are wasted and costs increase
Solution Approach 1:
The patent changes the parameters of compaction by introducing a cost model that evaluates multiple factors including I/O costs, computation costs, and storage efficiency. Instead of fixed-frequency compaction, the system dynamically adjusts compaction decisions based on real-time cost calculations, workload patterns, and storage conditions to achieve optimal balance between storage efficiency and resource consumption.
Solution Approach 2:
The patent implements feedback mechanisms by continuously monitoring system state, workload patterns, and compaction effectiveness. The cost model incorporates feedback from actual system performance data to refine future compaction decisions, allowing the system to learn from past actions and adjust strategies to minimize resource waste while maintaining storage efficiency.
2Quantity of substance
If traditional compaction techniques are used frequently, then data storage efficiency is improved, but system costs increase
Solution Approach 1:
The patent changes the parameters of compaction by introducing a cost model that evaluates multiple factors including I/O costs, computation costs, and storage efficiency. Instead of fixed-frequency compaction, the system dynamically adjusts compaction decisions based on real-time cost calculations, workload patterns, and storage conditions to achieve optimal balance between storage efficiency and resource consumption.
3Quantity of substance
If traditional compaction techniques are used, then data is compacted, but latency increases due to frequent operations
Solution Approach 1:
The patent applies dynamics by making the compaction strategy adaptive and flexible rather than static. The system dynamically adjusts compaction timing, intensity, and scope based on real-time conditions including workload patterns, storage conditions, and cost models. This allows the system to perform compaction only when and where it provides benefit, avoiding unnecessary operations that would increase latency.
Solution Approach 2:
The patent implements preliminary action by performing cost-benefit analysis and planning compaction operations in advance rather than executing them immediately. The system evaluates future costs and benefits of potential compaction operations, allowing for strategic timing that avoids peak latency periods while still achieving compaction goals.
4Quantity of substance
If traditional compaction techniques are used, then storage space is optimized, but I/O operations increase
Solution Approach 1:
The patent changes the parameters of compaction by introducing a cost model that evaluates multiple factors including I/O costs, computation costs, and storage efficiency. Instead of fixed-frequency compaction, the system dynamically adjusts compaction decisions based on real-time cost calculations, workload patterns, and storage conditions to achieve optimal balance between storage efficiency and resource consumption.
Data Source
AI summary
A compaction policy imposing soft limits to optimize system efficiency is used to select various rowsets on which to perform compaction, each rowset storing keys within an interval called a keyspace. For example, the disclosed compaction policy results in a decrease in a height of the tablet, removes overlapping rowsets, and creates smaller sized rowsets. The compaction policy is based on the linear relationship shared between the keyspace height and the cost associated with performing an operation (e.g., an insert operation) in that keyspace. Accordingly, various factors determining which rowsets are to be compacted, how large the compacted rowsets are to be made, and when to perform the compaction, are considered within the disclosed compaction policy. Furthermore, a system and method for performing compaction on the selected datasets in a log-structured database is also provided.