Parallel SVM Training with Distributed Kernel Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Support Vector Machines (SVMs) face significant challenges in scaling to large datasets due to high computation and storage requirements, which become impractically slow for problems involving a large number of training vectors, especially when trying to find the global optimum in high-dimensional spaces.
Innovation Solution
A parallel SVM technique is implemented by distributing kernel computation, kernel cache, and training data across multiple machines or processors, allowing each processing node to select a local working set, optimize, and update gradients until convergence, with a centralized function determining the global working set and updating gradients efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If SVM training is performed using traditional sequential methods, then classification accuracy can be achieved, but computation time and storage requirements become impractically slow for large numbers of training vectors
Solution Approach 1:
The training dataset is divided into multiple chunks or subsets that can be processed independently and in parallel across different computing nodes. Each chunk is trained separately using local SVM solvers, and the results are aggregated to form the global solution. This segmentation reduces the computational burden on individual processors and enables parallel processing, thereby improving training speed while managing computation and storage requirements effectively.
Solution Approach 2:
The problem is transformed from a single-machine sequential optimization problem into a multi-machine parallel optimization problem by introducing the dimension of distributed computing. The global optimization problem is decomposed into multiple local optimization problems that can be solved simultaneously across different computational nodes, effectively adding a spatial dimension to the computation process and reducing overall training time.
2Productivity
If the entire training dataset is stored and processed on a single machine, then complete data access is available, but storage costs and processing time become prohibitive for large datasets
Solution Approach 1:
The training dataset is partitioned into multiple chunks that are distributed across different computing nodes. Each node stores and processes only its local chunk, reducing individual storage requirements. The segmentation allows parallel processing of data chunks, improving overall processing efficiency while minimizing the storage burden on any single machine.
Solution Approach 2:
Each computing node independently processes its local data chunk using local SVM solvers, making self-service decisions about which local vectors become support vectors. This self-service approach eliminates the need for centralized data storage and processing, allowing each node to work autonomously with its local data portion while contributing to the global solution.
3Quantity of substance
If distributed processing is implemented across multiple machines, then computational and storage costs are reduced, but system complexity and coordination overhead increase
Solution Approach 1:
The system implements feedback mechanisms where local SVM solvers on distributed nodes report their results (local support vectors and optimization outcomes) to a central coordinator or master node. The coordinator aggregates these local results and uses feedback to guide further iterations of the distributed optimization process. This feedback loop enables coordinated optimization across multiple machines while maintaining manageable system complexity through structured information exchange.
Solution Approach 2:
The distributed processing system merges local optimization results from multiple computing nodes into a global solution. Local support vectors and optimization outcomes are combined through aggregation processes that produce the final global SVM model. This merging approach reduces system complexity by consolidating distributed computations into a unified result, managing coordination overhead through systematic combination of local solutions.
Data Source
AI summary
Disclosed is a parallel support vector machine technique for solving problems with a large set of training data where the kernel computation, as well as the kernel cache and the training data, are spread over a number of distributed machines or processors. A plurality of processing nodes are used to train a support vector machine based on a set of training data. Each of the processing nodes selects a local working set of training data based on data local to the processing node, for example a local subset of gradients. Each node transmits selected data related to the working set (e.g., gradients having a maximum value) and receives an identification of a global working set of training data. The processing node optimizes the global working set of training data and updates a portion of the gradients of the global working set of training data. The updating of a portion of the gradients may include generating a portion of a kernel matrix. These steps are repeated until a convergence condition is met. Each of the local processing nodes may store all, or only a portion of, the training data. While the steps of optimizing the global working set of training data, and updating a portion of the gradients of the global working set, are performed in each of the local processing nodes, the function of generating a global working set of training data is performed in a centralized fashion based on the selected data (e.g., gradients of the local working set) received from the individual processing nodes.