Partition Key Management for Database Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Partitioned database objects face inefficiencies due to uneven data distribution across partitions, leading to unbalanced usage and reduced throughput, as users struggle to predict data usage levels and maintain sorted order across multiple partitions, resulting in suboptimal performance and increased costs.
Innovation Solution
A client system automatically generates and manages partition key values to distribute data items across multiple partitions, ensuring balanced usage and maintaining sorted order within each partition, while dynamically rebalancing to adapt to changing access patterns and optimize the use of available throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If data items are stored in one partition to maintain sorted order, then sorting capability is improved, but throughput is reduced due to single partition access limits
Solution Approach 1:
The patent segments the sorted data set into multiple partitions, each maintaining local sorted order. The system divides the globally sorted requirement into locally sorted segments that can be accessed in parallel across multiple partitions, thereby achieving both sorting capability and improved throughput through parallel access.
Solution Approach 2:
The patent introduces a new dimension of organization by maintaining sorted order within each partition while adding partition-level metadata that enables efficient cross-partition queries. This multi-dimensional organization allows the system to achieve both local sorted access and global query capabilities simultaneously.
2Productivity
If data is stored across multiple partitions, then throughput capacity is improved, but aggregation work increases when querying across partitions
Solution Approach 1:
The patent performs preliminary organization of data within each partition to maintain sorted order before queries are executed. This pre-organization allows query results from multiple partitions to be efficiently merged and aggregated, reducing the computational overhead during actual query execution.
Solution Approach 2:
The system uses feedback from partition metadata and access patterns to optimize query execution. By tracking which partitions contain relevant data and their sorted state, the system can intelligently aggregate results with minimal work, adapting to changing data distributions and access patterns.
3Productivity
If partitions are kept balanced over time, then resource utilization is improved, but difficulty in predicting usage levels increases
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically monitors partition usage levels and performs rebalancing operations without external intervention. The system self-adjusts data distribution across partitions based on observed access patterns and usage metrics, eliminating the need for manual prediction and management of partition balance.
Solution Approach 2:
The patent introduces dynamic rebalancing that adapts to changing usage patterns in real-time. Rather than relying on static predictions, the system continuously adjusts partition assignments based on observed dynamics of data access and storage patterns, allowing optimal resource utilization even as usage levels change unpredictably.
4Stability of the object's composition
If single partition is used for sorted data, then sorted order is maintained, but access throughput is limited by partition throughput limit
Solution Approach 1:
The patent segments the sorted data across multiple partitions while maintaining sorted order within each segment. This segmentation allows parallel access to multiple sorted partitions simultaneously, achieving both the stability of sorted order and the productivity gain from increased access throughput through parallelization.
Solution Approach 2:
The patent creates a multi-functional partitioning system where each partition serves dual purposes: maintaining local sorted order for efficient range queries and enabling parallel access for high throughput operations. This universal approach allows the same partitioned structure to support both sorting requirements and high-performance access patterns.
Data Source
AI summary
Systems and methods are provided to implement a client system that distributes accesses to database object in a database across different partitions of the database object. The client specifies a partition key and a sort key for the database object, so that data items in the database object are partitioned based on the partition key and sorted in each partition based on the sort key. The client determines a set of partition key values based on a throughput criteria of the database object and a per-partition throughput limit. To insert a data item into the database object, the client assigns a partition key value from the set to the data item using an assignment function, configured to distribute accesses to the database object across its partitions to balance usage of the partitions. To obtain sorted data from the database object, the client issues queries to obtain results for each partition key value, and then merges the results to produce a fully sorted result set.