Subsurface Data Ingestion Reprocessing by Failure Stage Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Challenges arise in identifying and re-processing failed data records during large-scale data ingestion in subsurface data platforms due to computationally expensive tasks, where failures occur at various stages, making it difficult to pinpoint and re-process eligible records efficiently.
Innovation Solution
A recovery utility application monitors data ingestion operations, identifies failed records, and initiates re-processing requests for specific failure stages, using a failure message decoder to group eligible records for re-processing, thereby reducing computational overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all failed data records are re-processed through multiple processing stages, then data completeness is improved, but computational load and processing time increase significantly
Solution Approach 1:
The patent segments the large set of failed data records into smaller groups based on failure stage identifiers. Each group contains records that failed at the same processing stage, allowing targeted re-processing only for specific stages rather than re-processing all records through all stages, thus reducing computational load while maintaining data completeness.
Solution Approach 2:
The patent applies partial action by selectively re-processing only the groups of records that are eligible and necessary for re-processing, rather than re-processing all failed records. The system determines eligibility criteria and processes only those records that can potentially succeed, avoiding unnecessary computational expenditure on records that cannot be successfully re-processed.
2Measurement precision
If detailed tracking of each failed record is implemented, then failure identification accuracy is improved, but system complexity increases
Solution Approach 1:
The patent implements preliminary action by pre-grouping failed records into groups based on their failure stage identifiers before the re-processing decision is made. This pre-organization of data allows the system to efficiently identify which records failed at which stages without requiring complex real-time tracking during the re-processing phase, thus maintaining identification accuracy while reducing system complexity.
Solution Approach 2:
The patent creates a structured representation (copy) of the failure information by organizing failed record identifiers into groups associated with specific failure stage identifiers. This copied organizational structure enables efficient querying and processing without requiring the system to maintain complex relationships between individual records and multiple processing stages.
3Productivity
If selective re-processing of eligible records is implemented, then processing efficiency is improved, but risk of missing re-processable records increases
Solution Approach 1:
The patent implements feedback mechanisms where the system monitors the outcomes of re-processing operations and uses this information to identify patterns in failure and success. This feedback loop allows the system to refine its eligibility determination criteria over time, ensuring that selective re-processing does not miss potentially re-processable records while maintaining high processing efficiency through learned optimization.
Data Source
AI summary
Re-processing for subsurface data platform ingestion workflow services include processing multiple failure messages of a failure message file corresponding to a correlation identifier (id) identifying a data ingestion job, to obtain multiple groups corresponding respectively to multiple failure stage identifiers (ids). The re-processing further includes initiating a re-processing request for a first group of the multiple groups corresponding to a first failure stage id of the multiple failure stage ids, the re-processing request including the first group, the first failure stage id and the correlation id. The re-processing further includes re-processing the first group through a processing stage of the data ingestion job, the processing stage identified by the first failure stage id, to obtain a status of the re-processing request.