Data Integration Pipeline Using Differential Ingest and Import
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data integration methods become inefficient as data volume increases, requiring frequent reloading of entire databases for updates, leading to performance issues and inefficiencies.
Innovation Solution
A data integration pipeline is configured to receive and integrate raw data through two phases: ingest and import, using JSON file processing, parsing, and differencing to identify and exclude duplicative data, with object pointers for efficient retrieval and storage, optimizing data loading and reducing redundant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If entire database is reloaded for frequent updates, then data integration is achieved, but processing time and inefficiency increase
Solution Approach 1:
The patent segments the data integration process into distinct phases: ingest phase (parsing raw data into object fragments), difference phase (identifying changes between new and existing data), and import phase (applying only the differences). This segmentation eliminates the need to reload entire databases by processing only the changed portions, directly resolving the contradiction between achieving data integration and minimizing processing time.
Solution Approach 2:
The patent performs preliminary actions by parsing raw data into object fragments and storing them in a structured format before the actual import process. The difference phase pre-identifies which data objects have changed before the import phase begins. This preliminary preparation enables the system to skip processing unchanged data, improving productivity while reducing time loss.
2Reliability
If entire database is reloaded for large data volumes, then data updates are applied, but system performance deteriorates
Solution Approach 1:
The patent extracts only the necessary difference data from the complete dataset. The difference phase identifies and extracts only those object fragments that have changed between the new raw data and the existing database state. This extraction approach ensures data update completeness by capturing all changes while maintaining high system performance by excluding unchanged data from the import process.
Solution Approach 2:
The patent applies partial action by performing data integration operations only on the portion of data that has changed, rather than processing the entire database. The difference phase determines the exact subset of data requiring updates, and the import phase applies changes only to that subset. This partial processing approach maintains reliability by ensuring all necessary updates are applied while preserving system performance.
3Ease of operation
If duplicative data is included in reloads, then data integration is simplified, but storage efficiency and performance decrease
Solution Approach 1:
The patent uses object fragments as simplified copies of complete data objects. During the ingest phase, raw data is parsed into object fragments that contain only the essential information needed for comparison and import. These fragment copies enable easy identification of duplicative data through the difference phase while minimizing storage requirements, as only the necessary data portions are retained rather than complete duplicate objects.
Solution Approach 2:
The patent discards duplicative data by comparing object fragments in the difference phase and identifying which fragments represent unchanged data. These duplicative fragments are discarded from the import process. The system recovers only the necessary difference data that represents actual changes, thereby maintaining data integration simplicity while optimizing storage efficiency by eliminating redundant data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data integration pipeline is configured to receive and integrate raw data into a data processing system. Raw data may be defined as an electronic collection of data composed of information from multiple records, whose primary relationship to each other is their shared origin from a single or multiple databases. In integrating the raw data, the data integration pipeline may split the integration into two phases, more specifically: an ingest phase, and an import phase.