Incremental Key-Value Store Updates via TTL and Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large key-value database stores in production environments face inefficiencies in updating and maintaining data freshness due to the large size and high update rates, making daily updates unfeasible with current methods.
Innovation Solution
The system employs a database update system with a master database, changed database, and emergency database, utilizing a TTL mechanism and update modules to prioritize and incrementally update entries based on criticality and freshness, optimizing the update process by calculating a TTL value and refreshing sets to maintain data freshness while minimizing update times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all entries in a large key-value store are updated at high rate, then data freshness is improved, but update time becomes excessively long
Solution Approach 1:
The patent divides the large key-value store into multiple databases organized in a tree structure with root, intermediate, and leaf nodes. Each database contains a subset of entries, allowing incremental updates to be applied to smaller portions of the data store rather than requiring updates to all 100 million+ entries simultaneously. This segmentation enables the system to maintain data freshness for critical entries while avoiding the prohibitive time cost of updating the entire store.
Solution Approach 2:
The system performs partial updates by applying incremental changes to only the necessary subsets of databases rather than updating all entries. The multi-db architecture allows selective updates of specific leaf nodes or intermediate nodes based on which entries have changed, performing exactly the minimal necessary action to maintain freshness without the overhead of full-store updates.
2Loss of time
If incremental updates are applied to subsets of databases, then update time is reduced, but data consistency across the entire store may be compromised
Solution Approach 1:
The patent implements a nested database structure where leaf databases are contained within intermediate databases, which are in turn contained within the root database. Each level of the hierarchy manages a subset of entries, and updates propagate through the nested structure. This nesting ensures that data consistency is maintained across the entire store while allowing incremental updates at each level, as changes in leaf nodes are reflected in their parent intermediate nodes and ultimately in the root database.
Solution Approach 2:
The system uses TTL (time-to-live) mechanisms and version tracking to monitor data freshness and trigger updates. When entries in leaf databases exceed their TTL or when changes are detected, the system provides feedback to initiate incremental updates through the intermediate databases to the root database, ensuring consistency is restored without requiring full-store synchronization.
3Productivity
If the database is divided into multiple subsets, then update efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the large key-value store into multiple databases organized in a hierarchical tree structure. This segmentation improves update efficiency by allowing parallel or incremental updates to individual databases or subsets of databases without affecting the entire store. The modular structure enables independent management and updating of each segment, significantly improving productivity compared to monolithic update approaches.
Solution Approach 2:
The multi-db architecture serves multiple functions simultaneously: it enables incremental updates, provides data partitioning for scalability, implements TTL-based freshness management, and supports hierarchical data organization. This universal structure handles various operational requirements through a single cohesive system rather than requiring separate mechanisms for each function, thereby managing complexity efficiently.
Data Source
AI summary
Embodiments of methods, systems, and computer program products for incrementally updating a large key-value store can include a computer-implemented method of updating a master database comprising refreshing a refresh set of oldest entries in the master database, merging an emergency set of entries in an emergency database into the master database, and deleting the emergency set of entries from the emergency database. In some embodiments, the computer-implemented method can further comprise merging a changed set of entries in a changed database into the master database and deleting the changed set of entries from the changed database. Other embodiments of related methods and systems are also provided.