Primary-Key Table Segmentation for Faster Data Completion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data completion methods in database systems require creating column indexes for each column, leading to significant system resource consumption and long build times due to repeated scanning of main table data.
Innovation Solution
Perform area division on a target table based on a primary key column to obtain target areas, determine starting row offsets, and write data of target column groups to baseline sorted string tables using these offsets, eliminating the need for separate column indexes and reducing resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If column indexes are created for each column to enable data completion on column groups, then data completion can be performed, but system resource consumption increases significantly
Solution Approach 1:
The patent segments the table data into multiple area ranges based on the primary key column, and performs data completion on each area independently. This avoids the need to create column indexes for the entire table, reducing system resource consumption while maintaining data completion capability for each segment
Solution Approach 2:
The patent extracts only the necessary data from each area range and writes it to the corresponding baseline SSTable, rather than creating and maintaining column indexes for all columns. This extraction approach eliminates the overhead of index creation while achieving the same data completion objective
2Reliability
If column indexes are created for each column, then data completion can be performed on column groups, but the build time becomes long due to repeated scanning
Solution Approach 1:
By dividing the table into multiple area ranges based on primary key values, the patent enables parallel processing of data completion tasks. Each area can be processed independently and simultaneously, dramatically reducing the overall build time compared to sequential index creation across the entire table
Solution Approach 2:
The patent performs area division and determines starting row offset values before the actual data completion process. This preliminary preparation allows the main data completion task to proceed efficiently without repeated scanning and decision-making during the build phase, reducing total build time
3Productivity
If area division is performed on the target table based on the primary key column, then data can be written to baseline SSTable using starting row offsets, but the process complexity increases
Solution Approach 1:
The starting row offset value serves multiple functions: it identifies the area range, determines the starting position for data writing, and enables efficient data location in the baseline SSTable. This multi-functional parameter simplifies the overall process despite the initial area division step
Data Source
AI summary
The present application relates to the field of database technologies, and for example provides a data completion method and apparatus, an electronic device, and a storage medium. A data completion method includes: performing area division on a target table to be processed based on a primary key column in the target table to obtain at least one target area; determining a number of rows in each of the at least one target area; determining a starting row offset value of each target area based on the number of rows corresponding to each of the at least one target area; and writing data of each target column group of at least one target column group in the target table and in each target area to a baseline sorted string table corresponding to the target column group based on the starting row offset value of each target area. As such, it is not necessary to create corresponding column indexes for all columns separately, nor to sort data of each column through the column index, thereby simplifying the cumbersome steps of data completion and reducing consumption of system resources and time costs.