Ensemble Model Configuration for Inference Accuracy and Compute Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex deep neural networks require significant resources and time to generate inferences, making them challenging to handle large volumes of requests efficiently, while simplifying them often compromises inference accuracy.
Innovation Solution
A computing system evaluates and configures ensembles of machine-learning models to balance accuracy, compute resources, and latency, selecting the optimal ensemble for a given task type to ensure efficient and accurate responses within a specified time limit.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex deep neural networks are used to improve inference accuracy, then accuracy is improved, but compute resources and time required increase significantly
Solution Approach 1:
The patent divides a single complex deep neural network into multiple smaller sub-networks or model components that can be evaluated and executed separately. This segmentation allows the system to process inference requests through multiple parallel paths, reducing the computational burden on any single model while maintaining overall accuracy through ensemble evaluation.
Solution Approach 2:
The patent combines multiple smaller model evaluations into an ensemble approach where results from different model configurations are aggregated. By merging the outputs of several smaller models, the system achieves accuracy comparable to or exceeding that of a single large model while distributing computational resources across multiple smaller operations.
2Measurement precision
If complex deep neural networks are used to improve inference accuracy, then accuracy is improved, but time to generate inference increases
Solution Approach 1:
The patent segments the inference process into multiple smaller evaluation steps using different model configurations. Instead of running one time-consuming complex model, the system evaluates multiple smaller models in parallel or sequence, reducing the time required for each individual evaluation while maintaining accuracy through ensemble results.
Solution Approach 2:
The patent implements dynamic model selection where the system can adaptively choose which models to evaluate based on incoming request characteristics, resource availability, and accuracy requirements. This dynamic approach allows the system to optimize inference time by selecting appropriate model combinations for different scenarios rather than always using the most complex model.
3Use of energy by moving object
If smaller models are used to reduce compute resources, then compute resources are reduced, but inference accuracy decreases
Solution Approach 1:
The patent merges the results from multiple smaller model evaluations to achieve accuracy comparable to or exceeding that of larger models. By combining predictions from several smaller models through ensemble methods, the system compensates for the individual limitations of smaller models while maintaining low computational resource usage.
Solution Approach 2:
The patent creates a composite modeling approach where multiple smaller models with different strengths and characteristics are combined into an ensemble system. This composite structure leverages the diverse capabilities of individual smaller models to achieve overall performance that exceeds any single component, including larger models.
4Measurement precision
If ensembles of smaller models are used to maintain accuracy, then accuracy is maintained, but device complexity increases
Solution Approach 1:
The patent segments the model evaluation process into manageable components that can be independently configured and evaluated. This segmentation allows the system to handle complexity in a structured way, evaluating multiple model combinations through systematic exploration rather than monolithic complex processing.
Data Source
AI summary
Computer-assisted configuration of compute resource to perform tasks of a given inference task type. For each of multiple model combinations, the computing system estimates 1) a compute level that can perform tasks of the given inference type using the model combination, and 2) an accuracy of the model combination in performing tasks of the given inference task type. The computing system then selects a model combination for the given inference task type based on the estimated compute level of the model combination and the estimated accuracy of the model combination. In response to the selection, an inference component is configured to respond to task requests of the given inference task type by using the selected model combination. Scheduling using batch size and input size may further improve accuracy and efficiency of the model combination.