Unit of Work Key for Incremental Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data processing systems where only a small subset of data changes frequently, re-running programs on the entire dataset becomes impractical, and existing custom detection techniques struggle to support incremental updates, especially when multiple programs access the same data with unclear operations.
Innovation Solution
The implementation of a unit of work (UOW) key system that identifies and filters records based on a UOW key, allowing only modified records with matching UOW values to be processed, thereby reducing the data set that needs to be updated while ensuring accurate results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire dataset is re-processed to update results, then data accuracy is maintained, but processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the dataset into individual records with unique identifiers, allowing the system to process only the specific records that have changed rather than re-processing the entire dataset. This is achieved by identifying changed records through database triggers or change tracking mechanisms and then selectively re-processing only those segments.
Solution Approach 2:
The patent applies partial action by processing only the necessary subset of changed records rather than the complete dataset. The system identifies and processes only those records that have actual changes, avoiding unnecessary processing of unchanged records while maintaining data accuracy.
2Productivity
If custom detection techniques are implemented to identify changed records, then processing efficiency improves, but system complexity increases
Solution Approach 1:
The patent implements self-service by using database triggers that automatically detect and track changes to records. The database system itself performs the detection function through triggers that fire on insert, update, or delete operations, eliminating the need for external custom detection mechanisms and reducing overall system complexity.
Solution Approach 2:
The patent introduces an intermediary change tracking table that stores information about modified records. This intermediary structure simplifies the detection process by providing a centralized location to query for changed records, rather than implementing complex custom detection logic throughout the system.
3Loss of time
If incremental processing is implemented, then processing time is reduced, but ensuring data consistency across multiple programs becomes difficult
Solution Approach 1:
The patent creates a universal change tracking mechanism that serves multiple programs simultaneously. The change tracking table and trigger system provide a single source of truth about which records have changed, allowing any number of programs to consistently identify and process the same set of changed records without conflicts or inconsistencies.
Data Source
AI summary
Systems, methods, and other embodiments associated with a unit of work for incremental data processing are described. In one embodiment, a method includes selecting a unit of work key for a program that processes records in a source table. The unit of work key is selected such that modifications to a record having a certain unit of work key value will not affect the program's processing of records having a different unit of work key value. The selected unit of work key is associated with the program. When a record in the source table is created, modified, or deleted, a unit of work key value is identified for the record and a selected set of records having the identified unit of work key value is provided to the program. Thus, records in the data source that do not have the identified unit of work key values are not provided to the program.