Data Transformation Parallelization via Runtime Graph Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing number of processor cores in microprocessors is underutilized in business applications due to the requirement for specialized training and effort to write applications for parallel processing, which is not feasible for non-parallelizable applications.
Innovation Solution
A programming model and runtime architecture that automatically formulates data transformation processes for parallel execution without requiring dedicated parallelization coding, by classifying operations within a transformation graph and dynamically splitting and merging segments for parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If applications are written to take advantage of parallel processing threads, then processing power is improved, but programming effort and complexity increase significantly
Solution Approach 1:
The system automatically analyzes transformation graphs and identifies parallelizable operations without requiring programmer intervention. The runtime architecture self-determines which operations can be executed in parallel and distributes them across multiple processor cores, making the parallelization process self-service rather than requiring manual programming effort.
Solution Approach 2:
The system changes the execution parameters of operations by dynamically determining parallelization based on operation classification. Non-parallelizable operations are executed sequentially while parallelizable operations are executed concurrently, allowing the same code to adapt its execution parameters automatically based on the transformation graph analysis.
2Power
If additional processor cores are added, then processing power is improved, but utilization of additional cores decreases without specialized parallelization coding
Solution Approach 1:
The runtime architecture automatically detects and executes parallelizable operations across multiple cores without requiring specialized parallelization coding. The system self-determines which operations can be distributed to additional cores and executes them concurrently, maximizing core utilization automatically.
Solution Approach 2:
The system dynamically adjusts execution strategies based on the transformation graph analysis. At runtime, it determines which operations can be parallelized and assigns them to appropriate processor cores, allowing the system to adaptively utilize additional cores based on the specific operations being executed rather than requiring static parallelization coding.
3Productivity
If operations are classified for parallelization, then parallel execution efficiency is improved, but analysis and classification complexity increases
Solution Approach 1:
The system performs preliminary analysis of the transformation graph to identify parallelizable operations before execution. By classifying operations as parallelizable or non-parallelizable during the graph analysis phase, the system prepares the execution plan in advance, avoiding complex runtime decision-making and reducing overall system complexity.
Data Source
AI summary
One or more operations are defined to be used against a repository of one or more items. One or more associations between each of the one or more items are also defined. A classification is associated with the one or more operations for an execution environment within a transformation graph, wherein the classification defines whether the operation can be executed in parallel or not in parallel with other operations. The operations are processed based on the classification.