Distributed Database Joins Using Partial Replication and Redistribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed databases face the challenge of data skew, where some nodes process significantly more data than others, leading to slow overall processing and resource inefficiencies.
Innovation Solution
A data processing method and apparatus that performs partial replication, redistribution, and keeping unchanged operations on distributed data tables based on their distribution and join key characteristics to balance data load across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across multiple nodes in a distributed database, then system scalability and fault tolerance are improved, but data skew causes some nodes to process significantly more data than others, leading to slow overall processing
Solution Approach 1:
The patent segments data distribution strategies into three distinct types: partial replication for small tables, partial redistribution for skewed data, and partial keeping unchanged for already balanced data. This segmentation allows the system to apply the most appropriate strategy to each specific scenario, preventing data skew and improving overall processing speed by ensuring balanced node workloads
Solution Approach 2:
The patent implements partial operations rather than complete data movements. For example, partial replication replicates only necessary portions of data to specific nodes, partial redistribution redistributes only skewed data segments, and partial keeping unchanged maintains already-balanced data as-is. This partial action approach reduces unnecessary data movement overhead while effectively addressing data skew problems
2Productivity
If partial replication operation is performed on distributed data, then data balance across nodes is improved, but system complexity increases due to multiple operation types
Solution Approach 1:
The patent applies different data distribution strategies to different data scenarios locally. Small tables receive partial replication, skewed data receives partial redistribution, and already-balanced data receives partial keeping unchanged. This local quality approach allows the system to optimize each data segment independently with the most suitable strategy, improving processing efficiency without requiring complex global coordination
Solution Approach 2:
The patent performs preliminary analysis of data distribution characteristics before executing any data operations. By first identifying whether data is skewed, balanced, or requires replication, the system can pre-determine the appropriate operation type. This preliminary action prevents unnecessary data movements and simplifies subsequent execution by having a clear, predetermined strategy for each data segment
3Productivity
If data is evenly distributed across nodes, then query performance is improved, but data movement overhead increases during redistribution operations
Solution Approach 1:
The patent performs partial redistribution only on data segments that are actually skewed, rather than redistributing all data across the system. By identifying and targeting only the problematic data portions, the system achieves even distribution where needed while minimizing unnecessary data movement overhead and time consumption for already-balanced data segments
4Adaptability or versatility
If join operation is performed on distributed data tables, then data integration capability is improved, but processing time increases due to data skew on certain nodes
Solution Approach 1:
The patent segments the join operation processing by first analyzing data distribution characteristics of each table involved in the join. Based on this segmentation, it applies appropriate data balance operations (partial replication, partial redistribution, or partial keeping unchanged) to each table before executing the join. This ensures that join operations are performed on evenly distributed data, preventing time-consuming delays caused by data skew on specific nodes while maintaining full data integration capability
Data Source
AI summary
A data processing method of a distributed database (DDB) includes obtaining a first data table and a second data table that are to be joined in the DDB, first data in the first data table and second data in the second data table being distributed and stored in nodes; obtaining a first data distribution of the first data in the first data table and a second data distribution of the second data in the second data table; performing, based on the first data distribution and the second data distribution a partial replication operation, a partial redistribution operation, and a partial keeping unchanged operation, on full data including the first data and the second data, to obtain corresponding operated first data and operated second data; and obtaining a join result based on a join operation on the operated first data and the operated second data.