Data Transfer Job Error Handling via Task Short-Circuiting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data transfer systems face inefficiencies and errors during data transfer to external storage systems, particularly due to mis-configuration, system failures, and intermittent issues, leading to wasted resources and prolonged processing times.
Innovation Solution
A method is implemented where failed records are retried a predefined number of times, then logged for later attempts, and tasks are short-circuited if failure rates exceed a threshold, allowing for minimal reprocessing and efficient error handling in distributed processing systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If failed records are continuously retried indefinitely, then data transfer reliability is improved, but processing time and system resources are excessively consumed
Solution Approach 1:
The patent applies parameter changes by implementing a dynamic retry counter that tracks the number of retry attempts for each failed record. The system changes the retry parameter from indefinite to a predefined maximum number of attempts, and subsequently changes the state of failed records from active retry to error log storage after exhausting retries, thereby resolving the contradiction between reliability and processing time
Solution Approach 2:
The patent applies preliminary action by pre-defining the maximum retry count before the data transfer process begins. This preliminary parameter setup prevents the system from entering an infinite retry loop, allowing the system to proactively manage retry behavior and switch to error logging when the predefined limit is reached, thus balancing reliability with time efficiency
2Reliability
If retry attempts are increased to handle intermittent failures, then data transfer completeness is improved, but system resources are wasted on excessive retry operations
Solution Approach 1:
The patent applies parameter changes by implementing a dynamic retry counter that tracks the number of retry attempts for each failed record. The system changes the retry parameter from indefinite to a predefined maximum number of attempts, and subsequently changes the state of failed records from active retry to error log storage after exhausting retries, thereby resolving the contradiction between reliability and processing time
Solution Approach 2:
The patent applies the principle of cheap short-living objects by treating individual retry attempts as disposable resources. Instead of continuously investing system resources in potentially futile retry operations, the system allows a limited number of disposable retry attempts and then discards further retry efforts for records that have exhausted their retry budget, moving them to error logging instead
3Manufacturing precision
If all failed records are reprocessed from the beginning, then data accuracy is improved, but processing efficiency deteriorates due to redundant work
Solution Approach 1:
The patent applies segmentation by dividing failed records into distinct categories: records that are still within their retry budget and records that have exhausted their retry attempts. This segmentation allows the system to handle different types of failed records differently - continuing retries for some while logging errors for others, thereby maintaining data accuracy without forcing redundant reprocessing on all records
Solution Approach 2:
The patent applies taking out by extracting failed records from the main data transfer flow and placing them in a separate error log. This extraction removes problematic records that have exhausted their retry attempts from the active processing stream, allowing the main data transfer operation to continue efficiently while failed records are handled separately through error logging and potential manual intervention
4Loss of energy
If error logging is implemented for all failed records, then resource waste is reduced, but the ability to recover from temporary failures is diminished
Solution Approach 1:
The patent applies parameter changes by implementing a dynamic retry counter that tracks the number of retry attempts for each failed record. The system changes the retry parameter from indefinite to a predefined maximum number of attempts, and subsequently changes the state of failed records from active retry to error log storage after exhausting retries, thereby resolving the contradiction between reliability and processing time
Solution Approach 2:
The patent applies preliminary action by pre-defining the maximum retry count before the data transfer process begins. This preliminary parameter setup prevents the system from entering an infinite retry loop, allowing the system to proactively manage retry behavior and switch to error logging when the predefined limit is reached, thus balancing reliability with time efficiency
Data Source
AI summary
Disclosed are methods and apparatus for error handling within jobs that utilize a plurality of tasks for data transfer of individual data records to a storage destination. For each task, one or more failed records may be logged to a file for later insertion. If a high percentage of a task's output (e.g., writes to another data storage system) is determined to be failing, the task short-circuits itself. Each task is also configured to perform checkpoint logging as the task completes work. If the entire job later short-circuits and is to be restarted, the restarted job only repeats a minimal amount of previously completed work for the tasks which have not already completed their data insertions. Together, these techniques can ensure that in the face of periodic failures, the job completes long-running job in a minimal time with minimal effects.