Feature-Partitioned Decision Tree Training for Lower Communication Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing parallel distributed data training system for decision tree models consumes significant communication resources and memory overhead due to the transmission and storage of large gradient histograms during the training process.
Innovation Solution
The system divides the feature matrix into non-intersecting feature subsets among processing subnodes, allowing each subnode to determine local optimal splitting rules based on its subset, which are then transmitted to a main processing node for selection, reducing the need for global gradient histograms and minimizing network traffic and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If global gradient histograms are transmitted and stored during distributed decision tree training, then the model training can be performed, but communication resources and memory overhead increase significantly
Solution Approach 1:
The patent segments the feature matrix into multiple feature subsets, with each processing subnode receiving and processing only its assigned feature subset. This segmentation allows each subnode to independently compute local gradient histograms for its features without needing to transmit or store global gradient histograms, thereby reducing communication overhead while maintaining complete model training capability through aggregation of local results.
2Reliability
If global gradient histograms are transmitted and stored during distributed decision tree training, then the model training can be performed, but memory overhead increases significantly
Solution Approach 1:
The feature matrix is divided into multiple feature subsets distributed across processing subnodes. Each subnode computes gradient histograms only for its assigned features, storing only local gradient data rather than global gradient histograms. This segmentation dramatically reduces memory overhead while enabling complete model training through aggregation of local gradient information at the main processing node.
3Loss of energy
If feature matrix is divided into non-intersecting feature subsets among processing subnodes, then communication overhead is reduced, but the complexity of coordinating splitting rules increases
Solution Approach 1:
The feature matrix is segmented into non-intersecting feature subsets assigned to different processing subnodes. Each subnode independently determines local splitting rules for its features, and the main processing node aggregates these local rules to form global splitting decisions. This segmentation reduces communication overhead by eliminating the need to transmit complete gradient histograms, while the coordination complexity is managed through a clear division of labor between subnodes and main node.
Solution Approach 2:
The system implements a feedback mechanism where processing subnodes transmit their local splitting rule results to the main processing node, which then selects optimal splitting rules based on aggregated information from all subnodes. This feedback loop enables coordinated decision-making across the distributed system while maintaining independence in local computations, balancing communication efficiency with coordination requirements.
Data Source
AI summary
This application discloses a method to train a decision tree model. The method is performed by a training system. The training system includes N processing subnodes and a main processing node, N being a positive integer greater than 1. The method includes separately obtaining, by each processing subnode for a currently being trained tree node, a node training feature set and gradient data of the currently being trained tree node; separately determining, by each of the processing subnode, a local splitting rule for the currently being trained tree node according to the node training feature set and the gradient data that are obtained, and transmitting the local splitting rule to the main processing node; and selecting, by the main processing node, a splitting rule corresponding to the currently being trained tree node from the local splitting rule determined by each of the processing subnode.