Gradient Descent Training with Progressive Data Precision Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems face a dilemma in training with virtual data, where there is a tradeoff between the cost of high-precision and low-precision data points, leading to inefficiencies in gradient descent methods.
Innovation Solution
A cost-optimized gradient descent (COGD) method that iteratively adjusts parameters such as cost, gradient range, and learning rate to sequentially increase data precision, starting with low-precision virtual data and progressively refining it to achieve optimal training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision data points are used for training, then measurement precision is improved, but cost increases
Solution Approach 1:
The training process is segmented into multiple phases with different precision requirements. The patent divides training into stages where early phases use low-precision virtual data for coarse training, and later phases progressively use higher-precision data for fine-tuning. This segmentation allows the system to achieve high measurement precision without incurring the full cost of high-precision data throughout the entire training process.
Solution Approach 2:
The patent applies preliminary action by using low-precision virtual data for initial training phases before transitioning to high-precision data. The model is first trained coarsely with inexpensive virtual data to establish basic parameters, then progressively refined with higher-precision data. This preliminary training with cheaper data reduces the overall cost while still achieving the required measurement precision.
2Productivity
If gradient descent uses standard precision, then training speed is maintained, but cost efficiency deteriorates
Solution Approach 1:
The patent implements dynamics by making the gradient descent parameters adaptive rather than fixed. The learning rate, gradient clipping thresholds, and data precision levels are dynamically adjusted during training based on the current phase and model state. This allows the system to optimize both training speed and cost efficiency by using lower precision and smaller learning rates in early phases, then increasing precision and learning rate in later phases.
Solution Approach 2:
The patent applies parameter changes by systematically varying key gradient descent parameters throughout training. Learning rate is progressively increased, gradient clipping thresholds are relaxed, and data precision is increased in later phases. These parameter changes enable the system to maintain training speed while improving cost efficiency by avoiding the use of high-precision data and high learning rates throughout the entire training process.
3Measurement precision
If training starts with high precision, then measurement precision is improved, but cost increases
Solution Approach 1:
The patent applies preliminary action by using low-precision virtual data for initial training phases before transitioning to high-precision data. The model is first trained coarsely with inexpensive virtual data to establish basic parameters, then progressively refined with higher-precision data. This preliminary training with cheaper data reduces the overall cost while still achieving the required measurement precision.
Solution Approach 2:
The training process is segmented into multiple phases with different precision requirements. The patent divides training into stages where early phases use low-precision virtual data for coarse training, and later phases progressively use higher-precision data for fine-tuning. This segmentation allows the system to achieve high measurement precision without incurring the full cost of high-precision data throughout the entire training process.
Data Source
AI summary
Methods, systems and computer program products for training a network model (e.g. without limitation, a user transition probability model to mimic user interactions with a website) use a cost-optimized gradient descent (COGD) according to an embodiment. A method trains a model iteratively using COGD where, responsive to successive iterations of the training, parameters of the gradient descent are adjusted in a cost-optimized manner to sequentially increase data precision. Parameters of the gradient descent comprise may a cost parameter, a gradient range parameter and a learning rate parameter. By example, the cost is increased while either: gradient range is reduced; or gradient range and learning rate are reduced. In an embedment, goal-oriented training of the model is performed for each of a plurality of respective training goals, and training using cost-optimized gradient descent is performed in-turn (e.g. and in an order) for each respective training goal.