Gradient Boosting Model Updates With Orthogonal Residual Learners
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional gradient boosting tree algorithms face inefficiencies in online update processes due to the need for full data training, leading to an infinite enlargement of the prediction model scale and reduced computational efficiency when updating with stream data.
Innovation Solution
Implement orthogonal decomposition of gradient vectors to minimize correlations between base learners, allowing only a component perpendicular to existing base vectors to be fitted by new learners, expanding the model space and enabling learnable weighting coefficients, and updating weights using stream data for continuous online updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional gradient boosting tree algorithms are used for online updates, then the model can be updated with stream data, but the model scale will infinitely enlarge and computational efficiency will decrease
Solution Approach 1:
The patent extracts only the necessary gradient information from stream data rather than performing full data training. By taking out only the essential update components (gradient vectors and their orthogonal projections) from the streaming data, the model can be updated incrementally without incorporating unnecessary complexity, thus preventing infinite model scale enlargement while maintaining online update capability
Solution Approach 2:
The patent segments the gradient update process into orthogonal components that can be independently processed. By decomposing the gradient vector into orthogonal projections onto existing base vectors and perpendicular components, the update process is divided into manageable segments that can be applied incrementally to stream data, avoiding the need to retrain the entire model and thus controlling model scale growth
2Measurement precision
If full data training is performed for each update, then the model accuracy can be maintained, but the computational efficiency will be reduced
Solution Approach 1:
The patent performs preliminary orthogonal decomposition of gradient vectors during the offline training phase, establishing a basis of orthogonal projections before online updates. This preliminary action creates a computational framework that allows rapid online updates using only stream data, eliminating the need for computationally expensive full data training while maintaining model accuracy through the pre-established orthogonal basis
Solution Approach 2:
The patent changes the training approach from full data retraining to incremental parameter updates using only stream data. By updating model parameters through orthogonal gradient projections on streaming data rather than retraining on full datasets, the computational efficiency is dramatically improved while model accuracy is maintained through the mathematical properties of orthogonal decomposition
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
Embodiments of this specification provide service model training and online update methods and apparatuses. The service model can include several base learners, and the service model update method can include two parts: offline update and online update. During offline training of the service model, a training concept of increasing the quantity of base learners gradually based on a gradient boosting tree is used. For a newly added base learner used to fit a residual, only a component having a relatively small correlation with a known base learner is fitted by using orthogonal decomposition of a gradient. In an online prediction phase, a weight coefficient of each base learner can be updated by using stream data, and whether a new base learner needs to be added is detected. This implementation provides an effective solution for online update of a model under the gradient boosting tree.