Shared Execution of Mixed Data Flows in Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel processing systems, such as MapReduce-based platforms, face inefficiencies in processing large-scale data workloads due to redundant computations and scans across relational and non-relational operations, leading to increased processing and communication overhead.
Innovation Solution
The method involves identifying resource sharing opportunities across parallel tasks, including relational and non-relational operations, and sharing common data inputs, computations, intermediate results, and storage resources to optimize processing by designating a primary task and utilizing shared map and reduce functions to calculate cluster centers across multiple tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If each data flow operation is translated into separate MapReduce jobs executed independently, then task execution simplicity is maintained, but processing time and computational resources are wasted due to redundant scans and computations
Solution Approach 1:
The patent merges multiple independent MapReduce jobs into a unified execution framework where relational and non-relational operations share common data scans and computation resources. This combining eliminates redundant processing while maintaining the independence of individual operations, directly resolving the contradiction between execution simplicity and processing efficiency
Solution Approach 2:
The invention creates a universal execution engine that handles both relational operations (joins, groupings) and non-relational operations (clustering, classification) through shared infrastructure. This multi-functional approach allows a single system to perform diverse data processing tasks without requiring separate specialized jobs for each operation type
2Ease of operation
If multiple independent MapReduce jobs are executed for different data flow operations, then operational simplicity is maintained, but computational resources and communication overhead are increased
Solution Approach 1:
Multiple independent jobs are merged into a coordinated execution plan where common data scans are performed once and shared across multiple operations. This reduces computational resource consumption while maintaining operational simplicity through the unified management interface
3Reliability
If separate MapReduce jobs are used for each data flow operation, then job independence is maintained, but communication overhead and processing efficiency deteriorate
Solution Approach 1:
The patent combines multiple jobs into a single execution context where data flows are shared and coordinated. This merging reduces communication overhead by eliminating redundant data transfers between separate job executions while maintaining logical independence of operations through structured data flow management
Data Source
AI summary
A method, computer program product, and computer system for shared execution of mixed data flows, performed by one or more computing devices, comprises identifying one or more resource sharing opportunities across a plurality of parallel tasks. The plurality of parallel tasks includes zero or more relational operations and at least one non-relational operation. The plurality of parallel tasks relative to the relational operations and the at least one non-relational operation are executed. In response to executing the plurality of parallel tasks, one or more resources of the identified resource sharing opportunities is shared across the relational operations and the at least one non-relational operation.