Distributed Aggregate Reduction by Fragment Splitting in MLlib
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
MLlib, a machine learning framework in Spark, faces scalability issues due to a lack of object splitting functionality in its reduction step, limiting parallelism and processing performance.
Innovation Solution
A data processing method that splits aggregates into fragments, sends them to different execution workstations for reduction, and re-determines portions for further reduction until completion, enabling parallel processing across multiple workstations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If aggregate is processed as a single unit in reduction step, then implementation is simple, but parallelism is limited and processing performance is poor
Solution Approach 1:
The patent divides the aggregate into multiple aggregate fragments, allowing each fragment to be processed independently by different execution workstations. This segmentation enables parallel processing during the reduction step, significantly improving processing performance while maintaining manageable algorithm complexity through systematic fragment management
2Productivity
If aggregate is split into fragments for parallel processing, then parallelism is improved, but object splitting functionality is required which MLlib lacks
Solution Approach 1:
The patent extracts the object splitting functionality from the reduction step itself and implements it separately in the calculation step. By pre-splitting aggregates into fragments before reduction and managing the splitting logic independently, the solution enables parallel processing without requiring MLlib's reduction step to have native object splitting capabilities
3Productivity
If more aggregate fragments are created, then parallelism increases, but communication overhead between workstations increases
Solution Approach 1:
The patent combines multiple aggregate fragments through reduction operations across execution workstations in a coordinated manner. By merging fragments systematically and managing communication through structured data exchange protocols, the solution achieves high parallelism while minimizing communication overhead through efficient consolidation of results
Data Source
AI summary
A data processing method, an execution workstation, a distributed computing system and a computer-readable storage medium. The method includes: splitting an aggregate into a plurality of aggregate fragments; sending a first portion of aggregate fragments to a first execution workstation to perform reduction on the first execution workstation; receiving, from a second execution workstation, aggregate fragments produced by the second execution workstation and corresponding to a second portion of aggregate fragments and performing reduction on the aggregate fragments received and the second portion of aggregate fragments to update the second portion of aggregate fragments; re-determining a new first portion of aggregate fragments and a new second portion of aggregate fragments from a remaining portion excluding the first portion of aggregate fragments that have been sent from the plurality of aggregate fragments, and performing the sending, receiving and reduction; sending the aggregate fragment completing all reduction to a management workstation.