Parallel Index Rebuilding with Distinction Bit Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in index rebuilding, particularly during database recovery, due to the need for persistent index updates and high network overhead in index replication, which can lead to performance bottlenecks and resource consumption.
Innovation Solution
A parallel index rebuilding method using multi-core CPUs that compresses index keys by storing only distinction bits, allowing for faster index rebuilding by distributing data pages across cores, sorting compressed keys, and building the index tree in parallel, thereby reducing the need for disk I/O and network overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the index is rebuilt by storing and sorting complete index keys, then the index can be accurately reconstructed, but the memory space and processing time required increase significantly
Solution Approach 1:
The patent extracts only the distinction bits from complete index keys for storage and processing. Instead of retaining full index keys, the system identifies and stores only the critical bits that determine key ordering, significantly reducing memory space while preserving the ability to accurately reconstruct the index structure
Solution Approach 2:
The patent transforms the representation of index keys by changing from storing complete keys to storing compressed distinction bit representations. This parameter change in data representation enables the system to maintain index reconstruction accuracy while reducing the quantity of data that must be stored and processed
2Ease of manufacture
If the index is rebuilt sequentially using single-core processing, then the process is simpler to implement, but the rebuilding time increases significantly
Solution Approach 1:
The patent divides the index rebuilding process into segments that can be processed in parallel across multiple CPU cores. By partitioning the data pages and associated distinction bit extraction tasks across multiple processing units, the system achieves faster index reconstruction while maintaining implementation feasibility through modular design
Solution Approach 2:
The patent implements dynamic parallel processing where multiple CPU cores simultaneously perform distinction bit extraction and sorting operations on different portions of the index data. This dynamic utilization of multiple processing units reduces the overall rebuilding time compared to static single-core processing
3Reliability
If the complete index image is transmitted over the network to replicas, then the index can be replicated accurately, but the network overhead creates performance bottlenecks
Solution Approach 1:
The patent extracts and transmits only the distinction bits to replica systems rather than transmitting complete index images. This extraction of essential information reduces network bandwidth consumption and transmission time while providing sufficient data for replicas to reconstruct accurate copies of the index structure
Solution Approach 2:
The patent creates compressed copies of index information by representing full index keys as distinction bit patterns. These compressed copies require less network bandwidth to transmit to replicas while maintaining the fidelity needed for accurate index reconstruction at the replica systems
Data Source
Figure 1A~1B
Figure 2A~2C
Figure 3A~3B
AI summary
A system and method, including determining, by a processor, distinction bit positions for index keys of a database index; determining, by the processor, concatenation of the bits of the index keys; and generating a record of the concatenation. Other systems and methods include collecting, by a processor, index keys of a database index in parallel, wherein data pages of a target table are evenly distributed to a plurality of processor cores; scanning, by each of the cores the data pages to extract compressed keys and corresponding record identifiers; sorting, by a plurality of the processor cores, pairs of the compressed key and corresponding record identifier in accordance with a parallel sorting algorithm; generating an index tree in a bottom-up fashion; and storing a record of the generated index tree.