AI Model Weight Factorization for Low-Compute Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Updating AI models requires significant computation, which is costly and resource-intensive, especially when adapting to changing environments or user states.
Innovation Solution
The method involves obtaining a pre-trained AI model, quantizing weight values using first and second factor values, and updating the model based on training data by modifying first factor values while maintaining second factor values, reducing computation through optimized updating processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the AI model is updated frequently to adapt to changing environments and user states, then the adaptability and service quality are improved, but the computational cost and resource consumption increase significantly
Solution Approach 1:
The weight values are segmented into multiple factors (first factor values and second factor values). By dividing each weight value into components, the system can update only the first factor values while keeping second factor values fixed, reducing the computational burden while maintaining adaptability.
Solution Approach 2:
The patent changes the parameter representation of weight values from single values to factorized forms (products of first and second factor values). This parameter transformation enables selective updating of only certain factors, significantly reducing computational requirements for model updates.
2Productivity
If high-performance computing resources are provided to rapidly update the AI model, then the updating speed is improved, but the cost increases
Solution Approach 1:
By segmenting weight values into updateable first factors and fixed second factors, the system achieves rapid updates without requiring high-performance computing resources. The segmentation enables efficient computation by reducing the number of parameters that need to be recalculated.
Solution Approach 2:
The parameter factorization approach allows the system to achieve fast update speeds through mathematical optimization rather than brute-force computation, avoiding the need for expensive high-performance computing infrastructure.
3Measurement precision
If all weight values are updated to maintain model accuracy, then the model precision is improved, but the computational complexity increases
Solution Approach 1:
The segmentation of weight values into first and second factors allows the system to maintain model accuracy by updating only the necessary first factor values while keeping second factor values fixed, thereby reducing computational complexity without sacrificing precision.
Solution Approach 2:
By changing the parameter representation to a factorized form, the system can maintain model accuracy through updated first factors while avoiding the computational complexity of updating all weight values, achieving a balance between precision and complexity.
Data Source
AI summary
Provided is a method, performed by an electronic device, of updating a pre-trained artificial intelligence (AI) model may include obtaining a sum, of at least two first factor values to which at least two second factor values are respectively applied, as a quantized value of a first weight value from among a plurality of weight values included in the pre-trained AI model; obtaining training data for updating the pre-trained AI model; updating the pre-trained AI model based on the the training data.