Learned Index Storage Layout for Collision-Aware Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing learned index models face challenges in improving performance, particularly in managing dynamic data with complex distributions, leading to increased storage overheads and reduced efficiency in scenarios with massive data and limited memory.
Innovation Solution
The proposed method and apparatus utilize a learned index model where storage units can store either user data directly or a pointer to a collision array based on the quantity of data, employing a model algorithm to efficiently allocate data and reduce storage overheads while maintaining high-performance operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a learned index model is used to manage data, then data access performance is improved, but storage overheads increase
Solution Approach 1:
The patent segments the storage structure into two types: direct storage units for single records and collision arrays for multiple records. This segmentation allows the system to store data efficiently by choosing the appropriate structure based on data distribution, reducing unnecessary storage overheads while maintaining fast access performance.
Solution Approach 2:
The patent implements a dynamic storage mechanism where the learned index model can adaptively determine whether to store data directly or in collision arrays based on the quantity of data at each node. This dynamic adjustment optimizes storage overheads while preserving high data access performance.
2Reliability
If data is stored in collision arrays to handle multiple records, then data collision is resolved, but storage overheads increase
Solution Approach 1:
The patent applies local quality by creating collision arrays only where needed (when data collision occurs at a storage unit), rather than uniformly across the entire index structure. This allows the system to resolve data collisions reliably when necessary while minimizing storage overheads in regions where data is uniquely distributed.
3Productivity
If memory is limited, then storage capacity is constrained, but data processing efficiency must be maintained
Solution Approach 1:
The patent changes the storage parameter from fixed to adaptive, where the learned index model dynamically determines the storage structure based on data characteristics. This allows the system to maximize data processing efficiency within limited memory by optimizing the storage distribution according to actual data patterns.
Data Source
AI summary
A data processing method and apparatus are provided. The method includes: obtaining a key corresponding to to-be-read data; searching a learned index model for a leaf node corresponding to the key; and determining, according to a first model algorithm corresponding to the leaf node, a storage unit corresponding to the key. The storage unit corresponds to one or more pieces of user data. When the storage unit corresponds to a plurality of pieces of user data, the storage unit stores a first pointer pointing to a collision array. Additionally, or alternatively, when the storage unit corresponds to one piece of the user data, the storage unit stores the user data. The method further includes searching the collision array to which the first pointer points for the to-be-read data, or determining the user data stored in the storage unit as the to-be-read data.