Delta Index Alternation for Database Update Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in efficiently managing large database tables with millions of records, particularly in updating and searching data across indexes while minimizing hardware resource usage and maintaining performance.
Innovation Solution
The system employs multiple delta indexes that alternate in accepting updates, allowing main indexes to remain unchanged for longer periods, and uses dictionary-based compression with local and global value identifiers to facilitate fast searches and updates, enabling efficient data processing and optimization of memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a single main index that is updated frequently, then data availability is maintained, but hardware resources are heavily consumed and performance degrades
Solution Approach 1:
The patent divides the single main index into multiple delta indexes (first delta index and second delta index) that can alternate accepting updates. This segmentation allows the system to distribute update operations across multiple indexes, reducing the resource burden on any single index while maintaining data availability through redundancy.
Solution Approach 2:
The patent combines multiple delta indexes with a main index to form a unified data storage system. The delta indexes store recent updates while the main index maintains the baseline data, and they are merged during search operations to provide complete and up-to-date results without requiring continuous updates to a single index.
2Reliability
If the main index is rebuilt frequently to incorporate updates, then data consistency is maintained, but system downtime increases
Solution Approach 1:
The patent performs preliminary actions by maintaining delta indexes that continuously accept updates in the background. These delta indexes prepare the update data in advance, so when a main index rebuild is needed, the system can quickly merge the pre-prepared delta data without requiring lengthy update operations, thereby reducing system downtime.
Solution Approach 2:
The patent implements periodic action by alternating between multiple delta indexes for accepting updates. While one delta index is being used for updates, another can be merged with the main index. This periodic switching allows the system to maintain data consistency through regular merges while minimizing the frequency and duration of rebuild operations.
3Quantity of substance
If dictionary-based compression is used with local value identifiers in delta indexes, then memory footprint is reduced, but translation overhead is introduced
Solution Approach 1:
The patent introduces translation tables as intermediary components that map local value identifiers in delta indexes to global value identifiers in the main index. These translation tables act as mediators that enable the compressed storage structure to function correctly by providing the necessary translation layer without requiring fundamental changes to the compression approach.
Solution Approach 2:
The patent applies parameter changes by using different value identifier schemes for different indexes - local value identifiers for delta indexes to minimize memory usage, and global value identifiers for the main index for consistency. The translation mechanism dynamically adjusts between these parameter schemes as needed during merge operations.
Data Source
AI summary
Methods and apparatus, including computer program products, for selection of rows and values from indexes with updates. In general, rows of an index may be associated with validity flags that indicate whether a row has been updated with an update inserted in a delta index; one scheme for value identifiers may be used for an index and another scheme for one or more delta indexes where all of the indexes are, to at least some extent, compressed according to dictionary-based compression; and multiple delta indexes may be used in alternation such that one delta index may accept updates while another is being updated. The delta indexes may also have validity flags and all updates, such as modifications of values, deletion of records, and inserting of new records may be handled as updates accepted by one or more delta indexes.