Dynamic Row Merge in Columnar Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Columnar databases face inefficiencies in merging rows from a write store to a read store due to inflexible merging policies, which do not account for the likelihood of rows being updated, leading to increased computational costs from updating rows in the read store.
Innovation Solution
Implementing a dynamic merge criteria based on the modification age distribution, where rows are only merged to the read store if their current age exceeds a threshold age, determined using a set of elapse time buckets and statistical properties, to minimize unnecessary updates in the read store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If rows are merged from WRITE store to READ store based on a static policy (fixed time interval), then the merging process is simple to implement, but it leads to unnecessary updates in the READ store for rows that are still likely to be updated
Solution Approach 1:
The patent transitions from a static merging policy (fixed time intervals) to a dynamic policy that adapts to actual data characteristics. The system dynamically determines which rows to merge based on their update probability, using statistical analysis of modification patterns and age distributions. This dynamic approach resolves the contradiction by making the merging process intelligent and adaptive rather than rigid and uniform.
Solution Approach 2:
The patent changes the parameter used for merging decisions from a fixed time interval to a probabilistic criterion based on row age and modification patterns. By analyzing the distribution of modification ages and calculating update probabilities, the system dynamically adjusts merging criteria. This parameter change enables the system to avoid merging rows that are still likely to be updated, thereby reducing computational costs while maintaining data consistency.
2Ease of operation
If rows are merged from WRITE store to READ store based on a static policy, then the implementation is straightforward, but it overlooks differences among rows and merges rows that are still likely to be updated
Solution Approach 1:
The patent applies local quality by treating each row individually rather than uniformly. The system analyzes modification patterns and age distributions specific to each row or group of rows, determining their unique update probabilities. This localized analysis enables the merging decision to be tailored to the characteristics of each row, resolving the contradiction between operational simplicity and adaptability to row differences.
Solution Approach 2:
The patent introduces new parameters for analyzing row characteristics, such as modification age distribution and update probability. By changing from a single static parameter (time interval) to multiple dynamic parameters that capture row-specific behaviors, the system achieves both ease of operation through automated analysis and high adaptability to individual row differences.
3Productivity
If rows are merged into READ store that are still likely to be updated, then all rows are processed uniformly, but it increases computational costs to update rows in the READ store
Solution Approach 1:
The patent applies partial action by selectively merging only those rows that are unlikely to be updated, rather than merging all rows uniformly. By using statistical analysis to identify and filter out rows with high update probability, the system performs partial merging that avoids unnecessary computational expenses. This resolves the contradiction by achieving sufficient productivity through selective processing rather than exhaustive merging.
Solution Approach 2:
The system uses feedback from analyzing modification patterns and update histories to inform merging decisions. By continuously monitoring which rows are updated and when, the system builds statistical models that predict future update behavior. This feedback mechanism enables the system to make informed decisions about which rows to merge, balancing productivity with computational efficiency by avoiding rows that are likely to require updates.
Data Source
AI summary
Intelligent merging for efficient updates in columnar databases are disclosed, including: determining a current age associated with a row in a database, wherein the database comprises a write storage and a read storage, wherein the row is located in the write storage; determining a set of one or more merge criteria based at least in part on a modification age distribution, wherein the modification age distribution is dynamically determined based at least in part on previous transactions at the database; determining whether the current age meets the set of merge criteria; and in response to a determination that the current age meets the set of merge criteria, merging the row from the write storage into the read storage.