Gradient Boosted Trees Data Parallel Ensemble Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Gradient Boosted Trees (GBT) in machine learning are computationally expensive due to high I/O and communication costs in existing parallel implementations, which limit their performance.
Innovation Solution
Implementing data-parallel ensemble training for GBT by splitting the training dataset into portions assigned to thread groups, allowing each thread group to train and validate decision trees in parallel, with point-to-point data transfer and repeated iterations until a predetermined threshold is satisfied, resulting in faster execution and diverse tree ensembles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data parallel implementation is used for GBT, then training speed is improved through parallel processing, but I/O and communication costs increase due to collective reduction operations
Solution Approach 1:
The patent divides the training process into distinct segments: data parallel segmentation where the dataset is split across multiple processors, and tree ensemble segmentation where multiple tree ensembles are built independently. Each processor works on a portion of the data and builds local tree ensembles, avoiding the need for frequent collective reduction operations while maintaining parallel processing efficiency.
Solution Approach 2:
The patent performs preliminary actions by pre-splitting the dataset into partitions and pre-assigning them to different processors before training begins. This allows each processor to independently process its assigned data partition without requiring communication or coordination during the actual tree building process, thereby reducing I/O and communication costs.
2Productivity
If feature parallel implementation is used for GBT, then computational efficiency is improved through parallel processing, but communication overhead increases due to collective reduction operations
Solution Approach 1:
The patent segments the feature space by building multiple diverse tree ensembles that process different aspects of the data. Each ensemble processes features independently, and the final prediction is obtained by averaging the outputs of all ensembles. This segmentation approach allows parallel processing without requiring frequent communication or collective reduction operations between processors.
3Measurement precision
If sequential GBT training is used, then model accuracy is maintained through thorough data processing, but training time increases due to lack of parallelization
Solution Approach 1:
The patent divides the training task into multiple independent segments that can be processed in parallel. Each processor builds a complete tree ensemble on its assigned data partition, ensuring thorough processing similar to sequential training. The final model combines results from all segments, maintaining accuracy while achieving parallel speedup.
Solution Approach 2:
The patent creates multiple copies of the tree ensemble structure across different processors, with each processor building and maintaining a complete ensemble on its local data. This copying approach allows parallel processing of complete models rather than partial models, preserving the thoroughness of sequential training while enabling parallelization.
Data Source
AI summary
Data-parallel ensemble training using gradient boosted trees includes training an ensemble of trees. The training includes splitting a training dataset into several data portions. Each data portion is assigned to each thread group from a set of thread groups. The training further includes executing a stage, in which each thread group, in parallel, trains a respective ensemble of decision trees. Executing the stage includes performing, by each thread group, in parallel, machine learning operations for the respective ensemble of decision trees using the data portion assigned to each thread group. Further, each thread group validates, in parallel, the respective ensemble of decision trees using a data portion assigned to another thread group. Execution of the stage is repeated until a predetermined threshold is satisfied. Further, a prediction is inferenced using the ensemble of decision trees that is formed using the respective ensemble of trees from each of the thread groups.