Hybrid Job Processing for Bulk Data Extract
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data provisioning systems face challenges in reducing complexity and increasing efficiency when processing large volumes of data, particularly in migrating from mainframe solutions to centralized integrated data platforms, where serial processing is time-consuming and parallel processing loses dependencies, necessitating a more efficient method for handling bulk data.
Innovation Solution
The implementation of hybrid job processing, which involves modeling and transferring data from a raw data layer to a model data layer, utilizing a data model to determine dependencies and process tables in sequences where root tables are processed first, allowing for concurrent processing and minimizing processing time by over 70%.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If serial processing is used to maintain data dependencies, then data integrity is preserved, but processing time becomes excessively long (over 27 hours for half a billion records)
Solution Approach 1:
The patent segments the data processing workflow into distinct layers (raw data layer, modeled data layer, published data layer) and processes tables in sequences based on dependency relationships. This allows independent tables to be processed in parallel while maintaining proper execution order for dependent tables, thus reducing overall processing time while preserving data integrity.
Solution Approach 2:
The system dynamically determines processing sequences by analyzing data dependencies between tables. It automatically adjusts the processing order based on the specific dependency relationships in the dataset, enabling optimal parallel processing while ensuring that tables are processed in the correct sequence to maintain referential integrity.
2Productivity
If parallel processing is used to reduce processing time, then speed increases, but data dependencies between tables are lost
Solution Approach 1:
The patent segments tables into different processing sequences based on their dependency relationships. Tables with no dependencies can be processed in parallel across different sequences, while tables with dependencies are processed in the correct order within their sequence, thus maintaining both speed and dependency integrity.
Solution Approach 2:
The system introduces an intermediary layer (the dependency analysis and sequence determination mechanism) that mediates between parallel processing requirements and dependency constraints. This intermediary analyzes table relationships and orchestrates the processing order, enabling parallel execution where safe and sequential execution where necessary.
3Reliability
If complex dependency analysis is performed to maintain data relationships, then data integrity is improved, but system complexity increases
Solution Approach 1:
The system employs self-service mechanisms where the data model itself provides the dependency information needed for processing. By leveraging existing metadata and relationships defined in the data model, the system automatically determines processing sequences without requiring complex external analysis tools or manual configuration, thus maintaining integrity while limiting complexity growth.
Data Source
AI summary
Methods for hybrid job processing may include receiving raw data records stored within a plurality of tables from a plurality of systems of record at a raw data layer within a data exchange. Methods may include generating, based on a data model, a list of dependencies between the plurality of tables. Each table included in a second subset of the plurality of tables may be dependent on at least one table included in a first subset of the plurality of tables. Methods may include processing the first subset of the plurality of tables concurrently with one another. The processing includes modeling the raw data records and transmitting the modeled data records to the model data layer. Methods may include processing each table included in the second subset after completion of processing of the table included in the first subset from which the table in the second subset depends on.