Coordinator-Connector Data Ingestion for Failure Resume
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data ingestion systems are inaccurate and inefficient when processing data from third-party sources, often losing track of data status and wasting computational resources by reprocessing incomplete data due to system failures.
Innovation Solution
A sync coordination system uses a coordinator and connectors to manage data ingestion by setting a cursor location at failure points, storing unprocessed data in an object queue, and continuing the process in subsequent runs, thereby improving accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If watermarking is used to track data ingestion status, then data transfer progress can be monitored, but data accuracy is lost due to inability to track individual data item status
Solution Approach 1:
The patent segments the data ingestion tracking mechanism from the coarse-grained watermarking approach into fine-grained individual data item tracking. Each data item is independently monitored with its own status identifier, allowing precise tracking of each item's ingestion state (processed, failed, pending) while maintaining overall data transfer accuracy.
Solution Approach 2:
The patent implements a feedback mechanism where the status of each data item is continuously monitored and reported back to the system. This feedback loop enables the system to detect failed or incomplete data items and trigger appropriate remediation actions, ensuring both tracking precision and data transfer reliability.
2Productivity
If data ingestion is restarted after system failure, then data processing can resume, but computational resources are wasted by reprocessing already processed data
Solution Approach 1:
The patent performs preliminary action by recording the status of each data item before potential system failures occur. Status identifiers are maintained and persisted, so when the system restarts, it can immediately query these pre-recorded status values to determine which items need processing, avoiding reprocessing of already completed items and conserving computational resources.
Solution Approach 2:
The system uses feedback from status identifiers to intelligently resume data ingestion after failures. By querying the stored status information, the system receives feedback about which data items were successfully processed and which require reprocessing, enabling selective continuation that eliminates unnecessary computational waste.
3Reliability
If entire data pages are reprocessed to ensure completeness, then data accuracy improves, but processing efficiency decreases due to redundant operations
Solution Approach 1:
The patent segments the data validation approach from processing entire pages to validating individual data items. Each data item carries its own status identifier, allowing the system to selectively reprocess only the specific items that failed or were incomplete rather than reprocessing the entire page, thus maintaining data completeness while improving processing efficiency.
Solution Approach 2:
The patent applies partial action by reprocessing only the necessary portion of data (failed or incomplete items) rather than the entire data page. This selective reprocessing approach achieves the required data completeness accuracy while minimizing redundant operations and maximizing processing efficiency.
4Productivity
If data ingestion continues after partial page processing, then productivity increases, but data accuracy decreases due to loss of unprocessed data items
Solution Approach 1:
The patent performs preliminary action by capturing and storing the status of each data item before the system encounters failures or needs to pause processing. These status identifiers are persisted, enabling the system to later determine which items were processed and which were missed, ensuring data completeness while allowing continuous productivity during normal operation.
Solution Approach 2:
The system implements feedback through status identifiers that provide information about each data item's processing state. This feedback mechanism allows the system to continue processing new data items (maintaining productivity) while simultaneously tracking which items need attention, thereby preventing data loss and ensuring completeness through selective reprocessing when needed.
Data Source
AI summary
The present disclosure relates to systems, non-transitory computer-readable media, and methods for ingesting a dataset from a computer application that is external to a content management system. In particular, the disclosed systems can perform an ingestion process comprising a plurality of transfer runs by linking a content management system to the computer application with a connector. The disclosed systems can utilize a coordinator with computer logic to control the connector to determine a cursor location within a page of data at a failure point during a first transfer run. Moreover, the disclosed systems can store a subset of data from the page that comes after the cursor location and ingest the subset of data from the object queue by continuing the ingestion process according to the cursor location at the failure point of the first transfer run.