Log File Renaming for Key-Value Store Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern computing and storage systems face challenges in efficiently managing large volumes of key-value pairs across multi-tier storage facilities, particularly in high velocity and high volume data environments like IoT, where legacy techniques consume significant resources for data access and durability.
Innovation Solution
The technique involves renaming an existing log file to a data file and indexing its records, allowing key-value pairs to be accessed without copying data, thereby reducing memory, processing, and network resource demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If key-value pairs are stored in an in-memory storage tier using a binary search tree to facilitate low latency access, then data access speed is improved, but memory storage capacity is consumed and the tree depth grows reducing search efficiency
Solution Approach 1:
The patent segments the binary search tree into multiple smaller trees organized in a forest structure. Each tree in the forest manages a subset of key-value pairs, preventing any single tree from becoming too deep. This segmentation maintains search efficiency by limiting the maximum depth of individual trees while still providing low-latency access to all data.
Solution Approach 2:
The patent introduces a new dimensional organization by creating a forest of trees rather than a single tree. This adds a hierarchical dimension where trees are organized in levels, allowing the system to manage large volumes of data without proportionally increasing the depth of any individual access path. The forest structure provides an additional organizational layer that maintains search efficiency.
2Quantity of substance
If data is flushed from in-memory storage tier to on-disk storage tier to reclaim storage capacity, then memory storage capacity is recovered, but significant computing, networking, and storage resources are consumed
Solution Approach 1:
The patent extracts only the necessary metadata (tree structure information, node pointers) from the in-memory data and stores it on disk, while keeping the actual key-value data in memory. This selective extraction approach allows the system to maintain large datasets in memory without proportionally increasing the overhead of data management structures, thereby recovering effective storage capacity without consuming excessive resources for data movement.
Solution Approach 2:
The patent performs preliminary organization of data into forest structures and pre-calculates tree balances before potential flush operations. By preparing the data structure in advance with optimized tree configurations, the system minimizes the computational resources required during actual data flushing operations, as the heavy lifting of organization has already been completed.
3Quantity of substance
If the depth or breadth of the BST grows to manage large volumes of data, then storage capacity is increased, but search efficiency falls below acceptable threshold
Solution Approach 1:
The patent divides the large binary search tree into multiple smaller trees forming a forest, where each tree manages a subset of the total data. This segmentation ensures that no single tree becomes excessively deep, maintaining search efficiency within each tree while collectively providing large storage capacity across the entire forest structure.
Solution Approach 2:
The patent transitions from a single-dimensional tree structure to a two-dimensional forest structure with trees organized in levels. This dimensional change allows the system to scale storage capacity by adding more trees or levels rather than increasing the depth of individual trees, thereby maintaining search efficiency while accommodating large volumes of data.
Data Source
AI summary
Systems and methods for key-value pair management. A method embodiment commences upon receiving data records comprising key-value pairs. The key-value pairs are stored in two locations, (1) in an in-memory data structure in a first storage tier, and (2) in a log file in a second storage tier. At some moment in time, an event causes a flush of the in-memory data structure. The flush operations comprise renaming the log file to refer to it as a uniquely identifiable data file in the second storage tier, and populating an index file in the second storage tier. Requests for a value corresponding to a key is satisfied by using the index file to access the uniquely identifiable data file by byte or word offset to a location in the data file so as to retrieve the value of the corresponding key.