Differential Buffer for In-Memory Database Data Structures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ERP systems face inefficiencies in data manipulation and storage due to the need for reorganization of dictionaries and attribute vectors during data insertion and deletion, which increases computational complexity and time, especially in large datasets.
Innovation Solution
The introduction of a differential buffer that allows new data to be added as a new entry in the attribute vector, eliminating the need for rebuilding dictionaries and attribute vectors, and using separate encoding and decoding dictionaries to facilitate quick searching and data retrieval without the need for frequent sorting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If dictionary encoding is used with attribute vectors to organize data, then space requirements decrease and column scan speed increases, but inserting and deleting data becomes expensive due to reorganization requirements
Solution Approach 1:
The patent segments the attribute vector into multiple partitions or chunks, allowing insertions and deletions to occur locally within partitions without requiring complete reorganization of the entire attribute vector. This segmentation enables the system to maintain the benefits of dictionary encoding while reducing the complexity and cost of data manipulation operations.
Solution Approach 2:
The patent implements dynamic attribute vectors that can be efficiently updated through techniques such as lazy evaluation, deferred reorganization, and incremental updates. Instead of immediately reorganizing the entire attribute vector upon each insertion or deletion, the system dynamically manages reorganization operations to minimize their impact on performance while maintaining data integrity.
2Stability of the object's composition
If dictionaries are rebuilt during data insertion to maintain organization, then data structure integrity is maintained, but computational time and power increase
Solution Approach 1:
The patent performs preliminary actions by pre-allocating space in attribute vectors and dictionaries for future insertions, and by pre-establishing partition structures. This allows the system to accept insertions and deletions without immediate reorganization, thereby maintaining data structure integrity while avoiding the time cost of frequent complete rebuilds.
Solution Approach 2:
Instead of rebuilding dictionaries and attribute vectors after every insertion or deletion, the patent implements periodic reorganization where the entire structure is rebuilt only at scheduled intervals or when thresholds are reached. Between these periodic operations, the system uses incremental updates and local adjustments to maintain functionality, significantly reducing the average computational time required.
3Measurement precision
If frequent sorting is performed to maintain dictionary organization, then data retrieval accuracy is improved, but processing speed decreases
Solution Approach 1:
The patent applies sorting and reorganization only to local partitions or segments of the attribute vector that are affected by insertions or deletions, rather than sorting the entire structure. This local quality approach maintains retrieval accuracy for affected data while minimizing the processing speed impact by limiting the scope of sorting operations to small, manageable portions.
Solution Approach 2:
The patent dynamically adjusts the frequency and scope of sorting operations based on the current state of the data structure and query patterns. When the attribute vector is small or query patterns require high precision, more frequent sorting is performed. When the structure is large or query patterns are less sensitive to ordering, sorting is deferred or performed only on necessary partitions, thereby optimizing the balance between retrieval accuracy and processing speed.
Data Source
AI summary
A data structure and method of accessing, modifying, and adding data in the data structure are provided. The data structure includes an encoding dictionary, a decoding dictionary, and an attribute vector. The attribute vector includes information for cross-referencing characteristics associated with the same entry. Responsive to an instruction to add data, the method adds data to the encoding and decoding dictionaries as an entry and assigns a valueID for the added entries. The method then adds the valueID to attribute vector(s) associated with the dictionaries. Responsive to a query for data associated with a value, the method searches for the value in an encoding dictionary, determines a valueID corresponding to the data based on a match in the encoding dictionary, uses the attribute vector to find all relevant data across all dictionaries and attribute vectors, assembles the retrieved information in a tuple, then outputs all data associated with the value.