ML Model Inference Optimization via Pre-computed Normalization

Solution Overview

Problem

Machine learning systems for time-series forecasting in financial technologies require frequent updates and resource-intensive training, which is inefficient and costly, especially in applications like fraud detection and currency exchange.

Innovation Solution

Implementing a system with a training-platform auto-tuning process and inference-platform incremental updating process that uses GPU acceleration and unsupervised learning to optimize inference models and reduce resource consumption by automatically updating hyperparameters and generating optimized prediction responses.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If frequent model training is performed to maintain accurate time-series predictions, then prediction accuracy is improved, but computational resources and training time are excessively consumed

Engineering Contradiction:

Improveprediction accuracyVSAvoidcomputational resources

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary actions by pre-computing and storing statistics (mean, standard deviation) of input data during a training phase. These pre-computed statistics are then reused during inference to normalize new data points without requiring full model retraining, thereby reducing computational resources while maintaining prediction accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The model training and inference process is segmented into distinct phases: a training phase where full model training occurs with complete data processing, and an inference phase where only lightweight normalization using pre-computed statistics is performed. This segmentation allows the system to achieve accurate predictions during inference without consuming excessive computational resources.

Inventive Principle:

Principle #1Segmentation

2Reliability

If full model retraining is performed regularly to adapt to new data, then model accuracy is maintained, but training time and operational costs increase

Engineering Contradiction:

Improvemodel accuracyVSAvoidtraining time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-computing and storing statistics (mean, standard deviation) of input data during a training phase. These pre-computed statistics are then reused during inference to normalize new data points without requiring full model retraining, thereby reducing computational resources while maintaining prediction accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adapts between two operational modes: a training mode where the full model is retrained with new data to update weights and biases, and an inference mode where the model applies pre-computed normalization statistics to new data points. This dynamic approach allows the system to maintain accuracy while minimizing training time during regular operations.

Inventive Principle:

Principle #15Dynamics

3Measurement precision

If comprehensive data processing is performed during each prediction to maintain accuracy, then prediction quality is improved, but computational workload increases

Engineering Contradiction:

Improveprediction qualityVSAvoidcomputational workload

Core Design Contradiction:

Measurement precisionVSPower

Solution Approach 1:

The system performs preliminary actions by pre-computing and storing statistics (mean, standard deviation) of input data during a training phase. These pre-computed statistics are then reused during inference to normalize new data points without requiring full model retraining, thereby reducing computational resources while maintaining prediction accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates a simplified version of the data processing pipeline for inference: instead of performing comprehensive data processing during each prediction, it uses copied pre-computed normalization statistics (mean and standard deviation) to efficiently transform new input data. This copying approach maintains prediction quality while significantly reducing computational workload during inference.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS20230052255A1System and method for optimizing a machine learning model

Publication Date: 2023.02.16 VISA INTERNATIONAL SERVICE ASSOCIATION

AI summary

A machine learning system includes a training platform and an inference platform, where the inference platform is coupled to receive the output of the training platform. Based upon an updating of hyperparameters in the training platform, an optimized inference model is configured to be deployed to the inference platform from the training platform. The optimized inference model is further optimized in the inference platform by using an observation difference between a client observation and a prediction response to update the optimized inference model. The updated optimized inference model is used to provide a prediction response to a client.