Coordinator-Connector Data Ingestion After Transfer Failures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data ingestion systems are inaccurate and inefficient when processing data from third-party sources, often losing track of data processing status, resulting in incomplete data transfer and unnecessary computational resource waste.
Innovation Solution
A sync coordination system uses a coordinator and connectors to manage data ingestion by setting a cursor location at failure points, storing unprocessed data in an object queue, and continuing the process in subsequent runs, thereby improving accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing systems utilize watermarking to track data ingestion status, then they can establish reference points for processing subsequent pages, but they lose track of data and send incomplete or inaccurate data to internal data systems
Solution Approach 1:
The patent introduces a coordinator as an intermediary component that manages the ingestion process across multiple connectors. The coordinator maintains accurate tracking of data status by receiving and processing status information from connectors, ensuring that no data is lost or duplicated during the ingestion process from third-party systems to the internal data system.
Solution Approach 2:
The system implements a feedback mechanism where connectors report data status to the coordinator, which then uses this information to determine the next actions. The coordinator tracks which data has been successfully ingested and communicates this status back to the connectors, enabling accurate resumption of processing without losing track of data items.
2Reliability
If existing systems reprocess entire pages when encountering failures, then they ensure data completeness, but they waste computational resources on already processed data items
Solution Approach 1:
The patent segments the data ingestion process into individual data items or records within pages, rather than treating entire pages as atomic units. The coordinator tracks the status of each segment independently, allowing the system to resume processing only from the point of failure without reprocessing successfully ingested segments, thus reducing computational resource waste while maintaining data completeness.
Solution Approach 2:
Instead of reprocessing the entire page (excessive action), the system performs only the partial action necessary to complete the failed ingestion. The coordinator identifies exactly which data items need to be reprocessed and instructs connectors to process only those specific items, avoiding unnecessary reprocessing of already successful data while ensuring complete data ingestion.
3Productivity
If existing systems jump ahead to watermarks on subsequent pages, then they continue processing efficiently, but they skip or drop unprocessed data in previous pages
Solution Approach 1:
The coordinator maintains continuous feedback on the ingestion status of all data items across all pages. Before jumping ahead to process subsequent pages, the coordinator verifies that all previous data items have been successfully ingested. This feedback mechanism ensures that no data is skipped or dropped while maintaining efficient processing by only advancing when ready.
Solution Approach 2:
The system performs preliminary verification of data ingestion status before advancing to the next page. The coordinator checks whether all data items in current pages have been successfully processed and stored, and only then instructs connectors to proceed to subsequent pages. This preliminary action prevents data loss while maintaining processing efficiency.
Data Source
AI summary
The present disclosure relates to systems, non-transitory computer-readable media, and methods for ingesting a dataset from a computer application that is external to a content management system. In particular, the disclosed systems can perform an ingestion process comprising a plurality of transfer runs by linking a content management system to the computer application with a connector. The disclosed systems can utilize a coordinator with computer logic to control the connector to determine a cursor location within a page of data at a failure point during a first transfer run. Moreover, the disclosed systems can store a subset of data from the page that comes after the cursor location and ingest the subset of data from the object queue by continuing the ingestion process according to the cursor location at the failure point of the first transfer run.