AI Model Benchmarking via Module Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The development of artificial intelligence models requires extensive knowledge and resources to determine suitable hardware for efficient execution, as performance can vary significantly across different hardware platforms, leading to challenges in selecting the appropriate hardware for specific AI-based services.
Innovation Solution
A method involving a first computing device that receives module identification information from a second computing device to provide benchmark results, including performance information for entire models or configuration data for compressed models, to determine the optimal hardware node for executing AI-based models, considering factors like latency and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If comprehensive benchmark testing is performed across multiple hardware platforms, then hardware selection accuracy is improved, but time consumption and resource requirements increase
Solution Approach 1:
The patent pre-collects and stores benchmark data for multiple AI models across various hardware platforms in advance. When a user needs to select hardware for an AI model, the system directly retrieves pre-stored benchmark data instead of performing new comprehensive tests, significantly reducing time consumption while maintaining accurate hardware selection
Solution Approach 2:
The system performs comprehensive benchmark testing only for a limited set of representative AI models and hardware platforms initially. The collected benchmark data is then reused for similar models and comparable hardware, reducing the overall testing scope while still providing accurate hardware selection guidance for practical applications
2Measurement precision
If detailed performance metrics are collected for all AI models, then model-hardware compatibility assessment is improved, but data processing complexity increases
Solution Approach 1:
The patent extracts and focuses on collecting only the most critical performance metrics (such as inference time, memory usage, and computational load) that directly impact model-hardware compatibility. By filtering out redundant detailed metrics, the system maintains accurate compatibility assessment while reducing data processing complexity
Solution Approach 2:
The system collects detailed performance metrics specifically targeted at particular hardware characteristics and model requirements. Instead of uniformly collecting all possible metrics for all models, the benchmarking process adapts to collect only the relevant metrics for each specific model-hardware combination, reducing overall data complexity while maintaining assessment accuracy
3Measurement precision
If extensive benchmark data is stored for multiple models and hardware, then benchmark result accuracy is improved, but storage requirements increase
Solution Approach 1:
The system stores comprehensive benchmark data for a selective subset of representative AI models and hardware platforms. This partial data set is carefully chosen to cover the most common and important model-hardware combinations, providing accurate benchmark results for practical applications while avoiding the need to store exhaustive data for every possible combination
Solution Approach 2:
The patent designs the stored benchmark data to serve multiple purposes and model types. The benchmark results are structured to provide generalizable insights that can be applied across different but related models and hardware configurations, maximizing the utility of stored data while minimizing storage requirements through data reusability
Data Source
AI summary
Disclosed is a method performed by a first computing device performing a benchmark. The method may include receiving, from a second computing device comprising a plurality of modules which perform different operations related to an artificial intelligence-based model, module identification information indicating which module among the plurality of modules of the second computing device triggers a benchmark operation of the first computing device. The method may include providing, to the second computing device, a benchmark result based on the module identification information, and the benchmark result provided to the second computing device may be different according to the module identification information.