MPP Database Data Redistribution with Snapshot Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In massively parallel processing (MPP) databases, data redistribution processes are prone to failures due to server failures, leading to inefficient restarts and resource wastage, as individual instance failures can cause the entire process to fail, and maintaining isolation among operations is challenging.
Innovation Solution
A method for data redistribution in MPP databases that involves splitting job data into portions, recording a snapshot, and accumulating changes in a temporary table, allowing continuous processing even during redistribution, with failed redistributions being recoverable without losing data or processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If data redistribution is performed in MPP databases, then data can be evenly distributed across nodes improving load balance, but the process is prone to failures due to server failures requiring complete restarts
Solution Approach 1:
The patent segments the data redistribution process into independent instance-level operations. Each database instance can independently redistribute its portion of data without affecting other instances. This segmentation allows failed instances to be restarted independently rather than requiring complete process restart, directly addressing the reliability issue while maintaining data distribution stability.
Solution Approach 2:
The patent implements preliminary actions by creating snapshots and temporary tables before data redistribution begins. These preliminary structures serve as safety mechanisms that enable recovery if failures occur during redistribution, allowing the system to maintain stability even when reliability challenges arise during the redistribution process.
2Stability of the object's composition
If strict isolation is maintained among database instances during redistribution, then data integrity is preserved, but coordination complexity increases making continuous processing difficult
Solution Approach 1:
The patent segments the database system into independent instances that operate autonomously during data redistribution. Each instance maintains its own isolation boundaries while communicating through standardized snapshot and temporary table mechanisms. This segmentation preserves data integrity through isolation while reducing coordination complexity by eliminating the need for centralized control over entire redistribution processes.
Solution Approach 2:
The patent introduces snapshots and temporary tables as intermediary structures that facilitate communication and coordination between isolated instances. These intermediaries enable instances to share necessary information and maintain consistency without requiring direct complex coordination, thus preserving data integrity while managing coordination overhead.
3Stability of the object's composition
If complete restart is required upon redistribution failure, then data consistency is maintained, but processing time and computational resources are wasted
Solution Approach 1:
The patent performs preliminary actions by creating snapshots and temporary tables before redistribution begins. If failure occurs, these preliminary structures enable recovery without complete restart - the system can resume from the snapshot point using the temporary tables to track progress. This maintains data consistency through the safety mechanisms while avoiding the time and resource waste of complete restarts.
Solution Approach 2:
The patent implements a discard and recover mechanism where failed redistribution operations can be discarded at the instance level while other instances continue. The snapshot and temporary table structures enable recovery of successful portions without losing the work done by other instances, thus maintaining data consistency while minimizing time and resource loss.
4Stability of the object's composition
If individual instance failures cause entire process failure, then system-wide consistency is ensured, but productivity decreases due to complete process restarts
Solution Approach 1:
The patent segments the redistribution system into independent instances that can fail and recover independently. This segmentation maintains system-wide consistency through standardized snapshot and temporary table mechanisms while preventing individual failures from cascading to entire process failure. Instances can continue processing independently, maintaining productivity even when individual components fail.
Solution Approach 2:
The patent enables continuity of useful action by allowing successful instance redistributions to continue and remain valid even when other instances fail. The snapshot and temporary table mechanisms ensure that completed work is preserved and can be recovered, maintaining continuous productive operation across the system rather than requiring complete process interruption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.