Database Record Partitioning by Data Size
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in data partitioning, particularly in distributed environments, due to unequal distribution of data across partitions when partitioning by the number of change events, leading to uneven processing loads and reduced performance and stability.
Innovation Solution
Implementing size-based record partitioning, where the system determines average record sizes for each record type, assigns salted keys to create non-overlapping partitions of consistent sizes, ensuring uniform load distribution across processing nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is partitioned by the number of change events, then the partitioning process is simple, but the distribution of data across partitions becomes unequal leading to uneven processing loads
Solution Approach 1:
The patent changes the partitioning parameter from record count to data size (bytes). Instead of dividing partitions by number of change events, the system calculates total data size for each data schema type and creates partitions with equal byte distributions. This parameter change resolves the contradiction by maintaining partitioning simplicity while achieving balanced data distribution across partitions.
2Ease of manufacture
If data is partitioned by record count, then the partitioning method is straightforward, but processing performance decreases due to uneven partition sizes
Solution Approach 1:
The system changes the partitioning metric from record quantity to data volume in bytes. By calculating total bytes for each data schema type and distributing partitions based on equal byte ranges, the method maintains straightforward implementation while significantly improving processing performance through balanced partition sizes that prevent any single partition from becoming a bottleneck.
3Device complexity
If data is partitioned without considering record size, then the partitioning implementation is simple, but system stability decreases due to uneven load distribution
Solution Approach 1:
The patent introduces data size (bytes) as the partitioning parameter while keeping the implementation relatively simple. The system calculates total data size for each data schema type, determines partition byte ranges, and assigns records to partitions based on their size-weighted key distributions. This approach maintains acceptable implementation complexity while dramatically improving system stability through balanced load distribution across all partitions.
Data Source
AI summary
A database system is disclosed. The database system may receive a plurality of records representing changes made to a plurality of datasets, and the plurality of records may include a plurality of data change instances associated with a plurality of data schema types. The plurality of data change instances may correspond to a plurality of respective keys. The database system may compute an average size of data for each data schema type of the plurality of data schema types. The database system may modify the plurality of respective keys through a salting process, which may include assigning non-overlapping numerical ranges to subsets of the plurality of respective keys. The database system may sort, using the non-overlapping numerical ranges, the plurality of data change instances into a plurality of partitions of a target file size. The database system may write files of the target file size to a data store.