Cascaded Data Object Processing for Parallel Resource Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large data processing systems, the sequential selection of data objects in the main process determines the total processing time, leading to inefficient use of parallel processing resources, as the distribution of data between accounts is unknown, and existing methods either result in idle processors or produce numerous separate result logs that are difficult to evaluate.
Innovation Solution
A cascaded processing method where a main process distributes folders to parallel processes, which select and split data objects into packages, further distributing them to additional parallel processes, allowing for dynamic and automatic distribution to minimize processing time and optimize resource use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential selection of data objects is performed in the main process, then data objects can be selected and processed, but the total processing time is determined by the sequential selection step, leading to inefficient use of parallel processing resources
Solution Approach 1:
The patent segments the data processing task into two distinct phases: a selection phase that distributes folders to parallel processes, and a processing phase where multiple parallel processes handle data objects simultaneously. This segmentation allows the sequential bottleneck to be isolated to only the folder distribution step, while the actual data processing occurs in parallel across multiple processes, thereby improving overall productivity without excessive time loss.
Solution Approach 2:
The patent introduces a hierarchical dimension to the processing architecture by organizing processes into tiers: the main process distributes folders to first-tier parallel processes, which then distribute data objects to second-tier parallel processes. This multi-dimensional organization allows simultaneous execution at multiple levels, transforming the single-threaded sequential bottleneck into a multi-level parallel system that充分利用 processing resources.
2Productivity
If data is divided into packages for parallel processing, then processing can be distributed, but without a priori criteria for division, it is unclear how to optimally distribute data between accounts and processes
Solution Approach 1:
The patent implements self-service by allowing each first-tier parallel process to autonomously select and distribute data objects from its assigned folder to second-tier processes. Instead of requiring centralized control or complex pre-computed distribution plans, each process independently manages its own data distribution based on local conditions, simplifying the overall system while maintaining parallel processing efficiency.
Solution Approach 2:
The patent introduces dynamic adaptability by allowing the data distribution strategy to adjust based on runtime conditions. Each parallel process can dynamically determine package sizes and distribution patterns based on the actual data characteristics it encounters, rather than relying on static pre-defined schemes. This dynamic approach optimizes parallel processing efficiency without requiring complex upfront planning.
3Productivity
If separate batch processes are started for each account, then parallel processing can be maximized, but a very large number of separate result logs are produced, making evaluation difficult
Solution Approach 1:
The patent merges the results from multiple parallel processes into a single consolidated result log. Instead of maintaining separate logs for each account or process tier, the system combines all processing outcomes into one unified output that can be easily evaluated. This merging approach maintains high processor utilization through parallel execution while eliminating the complexity of managing and reviewing numerous separate result files.
Data Source
AI summary
Systems and methods are provided for cascaded processing a plurality of data objects. Each data object may be assigned to a folder among a plurality of folders, wherein a main process distributes the plurality of folders to a corresponding number of parallel processes of a first tier of processes. Each parallel process may select the data objects to be processed for the respective folder, split the selected data objects into packages of appropriate size, respectively, and distribute the packages similar to a cascade to a corresponding number of further parallel sub-processes of a second tier of processes, whereupon each package is processed.