Gradient Boosting Decision Tree Categorical Feature Enhancement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Gradient boosting decision trees face challenges in maintaining accuracy and reducing training time, particularly with categorical variables, as they typically use a single threshold as a split point, leading to overfitting and inefficient resource utilization.
Innovation Solution
The method involves sorting feature values by their gradient during training, performing a linear search to find a subset with maximum split gain, and modifying nodes to have multiple split points, while combining low population feature values into virtual bins to prevent overfitting and improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If multiple split points are used on nodes for categorical features, then manufacturing precision (model accuracy) is improved, but device complexity (tree structure complexity) increases
Solution Approach 1:
The patent segments categorical features by identifying multiple split points within a single node, dividing the feature space into multiple segments. This allows the model to capture complex categorical relationships without creating separate nodes for each category, thus improving accuracy while controlling tree structure complexity.
Solution Approach 2:
The patent introduces a new dimension to the traditional decision tree structure by allowing multiple split points on the same node for categorical features. This dimensional change enables the model to represent complex categorical relationships without increasing the tree depth or node count, resolving the contradiction between accuracy and complexity.
2Manufacturing precision
If multiple split points are used on nodes, then manufacturing precision (model accuracy) is improved, but loss of time (training time) increases
Solution Approach 1:
The patent performs preliminary actions by pre-identifying and storing multiple split points for categorical features during the training process. This allows the model to quickly evaluate multiple categories without repeatedly scanning the data, thus improving accuracy while reducing the time cost during training and inference.
3Manufacturing precision
If multiple split points are used on nodes, then manufacturing precision (model accuracy) is improved, but productivity (resource efficiency) decreases
Solution Approach 1:
The patent segments categorical features into multiple split points within a single node, allowing efficient resource utilization by evaluating multiple categories without creating additional tree nodes. This segmentation approach improves model accuracy while maintaining resource efficiency during inference.
4Device complexity
If single threshold split point is used, then device complexity (tree structure) is reduced, but measurement precision (accuracy on categorical variables) deteriorates
Solution Approach 1:
The patent introduces a new dimension to the decision tree structure by allowing multiple split points on the same node for categorical features. This enables the model to capture complex categorical relationships without increasing tree depth or node count, thus maintaining simple tree structure while improving measurement precision on categorical variables.
Data Source
AI summary
A computer implemented method of generating a gradient boosting decision tree for obtaining predictions includes finding split points by sorting variable values of a feature by their gradient during training of the gradient boosting decision tree, performing a linear search to find a subset of variables with maximum split gain, and modifying a node of the gradient boosting decision tree to have multiple split points on the node for a feature as a function of the linear search. In a further example, a computer implemented method of controlling overfitting in a gradient boosting decision tree includes combining values of low population feature values into a virtual bin, fanning out the virtual bin into feature values having a low population, and including the low population feature values into multiple split points on a node of the gradient boosting decision tree.