Parallel Data Processing Framework for Incremental Batch Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data processing systems face inefficiencies due to the need to reprocess entire data sets each time, leading to high latency and maintenance complexities, especially when only minor changes occur, as they operate in batch mode rather than incremental mode.
Innovation Solution
A parallel data processing framework that transforms synchronous code into a pipeline application capable of executing in both batch and incremental modes, allowing for the identification and processing of differences between data sets, thereby reducing processing time and eliminating the need for separate batch and incremental systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch processing is used to process entire data sets, then completeness of processing is ensured, but processing time increases and latency increases
Solution Approach 1:
The patent segments the data processing task by dividing the data set into processed and unprocessed portions. The system identifies and processes only the incremental changes (new or modified records) rather than reprocessing the entire data set, thus segmenting the work to eliminate redundant operations while maintaining processing completeness.
Solution Approach 2:
The patent applies partial action by processing only the necessary portion of the data set - specifically the incremental changes since the last processing run. This avoids excessive processing of already-updated records, reducing processing time while ensuring all necessary updates are applied to maintain output completeness.
2Adaptability or versatility
If separate batch and incremental systems are maintained, then specific processing needs are met, but system complexity increases
Solution Approach 1:
The patent implements a universal processing system that can operate in both batch and incremental modes through a single codebase. The system automatically detects whether to process entire data sets or only incremental changes based on the presence of change tracking mechanisms, eliminating the need for separate batch and incremental systems while maintaining all necessary processing capabilities.
Solution Approach 2:
The patent merges batch and incremental processing capabilities into a single unified system. By combining change tracking functionality with the existing processing pipeline, the system can dynamically switch between processing modes without requiring separate maintenance tracks, thus reducing system complexity while preserving adaptability.
3Reliability
If entire data sets are reprocessed regularly, then data freshness is maintained, but compute resources are wasted
Solution Approach 1:
The patent applies preliminary action by implementing change tracking mechanisms that identify which data records have changed since the last processing run. This preliminary identification of changes allows the system to prepare a targeted processing list, avoiding the waste of compute resources on unchanged records while ensuring data freshness is maintained through timely processing of updates.
4Ease of manufacture
If batch processing is used, then simplicity of implementation is maintained, but processing frequency is limited
Solution Approach 1:
The patent introduces dynamics by enabling the processing system to adapt its behavior based on real-time conditions. The system can dynamically switch between batch and incremental processing modes, and can be triggered by events such as data changes or time intervals, allowing more frequent processing runs without increasing implementation complexity significantly, as the same codebase handles both modes.
Data Source
AI summary
One example method includes identifying synchronous code including instructions specifying a computing operation to be performed on a set of data; transforming the synchronous code into a pipeline application including one or more pipeline objects; identifying a first input data set on which to execute the pipeline application; executing the pipeline application on a first input data set to produce a first output data set; after executing the pipeline application on the first input data set, identifying a second input data set on which to execute the pipeline application; determining a set of differences between the first input data set and second input data set; and executing the pipeline application on the set of differences to produce a second output data set.