Histogram-Based Decision Tree Training With Hybrid CPU–GPU Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing decision-tree based machine learning models are resource and time-consuming due to inefficient tree building algorithms, particularly in methods like random forest and gradient boosting, which lack scalability and speed in training processes.
Innovation Solution
A hybrid tree building algorithm utilizing a breadth-first search (BFS) on a graphics processing unit (GPU) with concurrent depth-first search (DFS) on a central processing unit (CPU), leveraging histogram-based compressed representations and stream processing to map tree building operations across multiple GPU streams, optimizing node splits and histogram updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional CPU-based tree building algorithms are used, then implementation simplicity is maintained, but training speed and scalability are limited
Solution Approach 1:
The patent segments the tree building process into independent operations that can be executed in parallel on GPU streams. Each stream processes a subset of nodes or operations independently, allowing concurrent execution of multiple tree building tasks. This segmentation enables the system to leverage GPU parallelism while maintaining algorithmic correctness through proper synchronization.
Solution Approach 2:
The patent introduces histogram-based compressed representations as an intermediary data structure between the raw training data and the tree building algorithm. This intermediary representation compresses the training examples into histogram bins, reducing memory access requirements and enabling faster parallel processing on GPU while preserving the necessary information for accurate tree construction.
2Productivity
If GPU-based parallel processing is implemented, then training speed increases, but resource consumption and implementation complexity increase
Solution Approach 1:
The patent applies partial action by selectively parallelizing only the computationally intensive histogram update and node split operations on GPU, while keeping other operations on CPU. This selective parallelization achieves significant speedup for the bottleneck operations without unnecessarily complicating the entire system or consuming excessive resources for operations that don't benefit from GPU acceleration.
Solution Approach 2:
The patent changes the representation parameters of the training data by compressing continuous feature values into discrete histogram bins. This parameter transformation reduces the complexity of parallel processing by converting continuous data into discrete categories, enabling efficient GPU-based histogram operations while reducing memory bandwidth requirements and computational resource consumption.
3Loss of time
If comprehensive tree building operations are performed on GPU, then processing speed increases, but memory requirements and data transfer overhead increase
Solution Approach 1:
The patent extracts only the essential features needed for tree building into compressed histogram representations, leaving detailed raw data on CPU memory. This extraction approach allows the GPU to process condensed feature summaries with minimal memory footprint, reducing both GPU memory requirements and data transfer overhead while maintaining training accuracy through the preserved statistical information in histograms.
Data Source
AI summary
According to one embodiment, a method, computer system, and computer program product for training a cognitive model that involves one or more decision trees as base learners is provided. The present invention may include constructing, by a tree building algorithm, the one or more decision trees, wherein the constructing further comprises associating one or more training examples with one or more leaf nodes of the one or more decision trees and iteratively running a breadth-first search tree builder on one or more of the decision trees to perform one or more tree building operations; and training the cognitive model based on the one or more decision trees.