Database Partitioning Minimizing Distributed Transaction Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face scalability issues due to inefficient partitioning of data and workload across nodes, leading to increased overhead from blocking, inter-node communication, and load balancing problems, which negatively impact throughput and performance in high-concurrent transaction processing applications.
Innovation Solution
A method and system for partitioning databases by grouping entries into cohesive units and dividing them into partitions to minimize the number of transactions accessing multiple partitions, utilizing a graph structure to optimize the distribution of transactions across nodes, thereby reducing distributed transactions and improving system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data and workload are partitioned across multiple shared-nothing nodes, then system scalability is improved, but the number of distributed transactions accessing multiple partitions increases, leading to increased overhead from blocking, inter-node communication, and load balancing
Solution Approach 1:
The patent segments the database into partitions based on entry groups that are accessed together by transactions. By dividing the database schema into multiple partitions and routing transactions to specific partitions based on their access patterns, the system reduces the number of distributed transactions that need to access multiple partitions, thereby reducing overhead while maintaining scalability.
Solution Approach 2:
The patent performs preliminary analysis of transaction patterns to identify entry groups that are always accessed together. This preliminary action allows the system to pre-determine the optimal partitioning scheme and transaction routing strategy before actual operations, minimizing the need for complex runtime decision-making and reducing distributed transaction overhead.
2Productivity
If entries accessed together are stored on the same node, then the number of distributed transactions is minimized, but data and workload skew may occur across nodes
Solution Approach 1:
The patent applies local quality by creating different partitioning strategies for different parts of the database schema based on their access patterns. Each partition is optimized for its specific workload characteristics, allowing entries accessed together to be stored on the same node while maintaining overall system balance through targeted rather than uniform partitioning.
Solution Approach 2:
The patent changes the parameter of transaction routing by using a routing table that maps transactions to specific partitions based on analyzed access patterns. This parameter change allows the system to dynamically adjust how transactions are distributed across nodes, minimizing distributed transactions while preventing excessive skew through intelligent routing decisions.
3Productivity
If an optimal database partition design is implemented, then throughput and performance are improved, but the complexity of determining entry groups and transaction routing increases
Solution Approach 1:
The patent implements self-service by automatically analyzing transaction patterns and entry access relationships to determine the optimal partitioning scheme. The system performs self-analysis to identify entry groups and generate routing tables without requiring manual intervention, thereby improving throughput while managing design complexity through automation.
Solution Approach 2:
The patent uses feedback mechanisms by analyzing actual transaction patterns and access relationships to refine the partitioning design. The system continuously monitors how transactions access entries and uses this feedback to optimize the routing table and partition configuration, improving throughput while keeping the design process manageable through data-driven iterations.
Data Source
AI summary
The present invention relates to a method and system for partitioning a database. The method for partitioning a database comprises: grouping a plurality of entries in the database into one or more entry groups, so that entries in the same entry group are always accessed together by one or more transactions; and dividing the one or more entry groups into a set number of partitions, so that a total number of transactions that access across more than one partition is minimized. By means of the present invention, it is possible to obtain an efficient, flexible and convenient method for partitioning a database, thereby greatly improving the system performance.