Model Maintenance Device for Pattern Recognition Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional pattern recognition systems face challenges in determining whether to update a model without compromising recognition performance for specific types of data, as simplistic performance evaluations can lead to declines in recognition rates for critical data types, even if overall performance improves.
Innovation Solution
The system classifies evaluation data into multiple groups, calculating separate performance evaluation values for each group to determine if the existing model should be updated based on threshold values, ensuring that recognition performance for critical data types is maintained.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a new model is updated based on overall performance evaluation value, then overall recognition performance is improved, but recognition rate for specific data types may decline
Solution Approach 1:
The evaluation data set is divided into multiple groups based on data types, and performance evaluation values are calculated separately for each group. This segmentation allows the system to evaluate both overall performance and specific data type performance independently, preventing updates that would harm critical data types while still improving overall performance.
Solution Approach 2:
Different evaluation criteria and threshold values are applied to different data type groups. Critical data types can have stricter performance requirements and higher threshold values, while less critical types allow more flexibility. This local quality approach ensures that updates maintain acceptable performance across all data types while enabling overall improvement.
2Reliability
If model update is restricted to maintain specific data type recognition rate, then reliability for critical data is maintained, but overall productivity improvement is limited
Solution Approach 1:
The system dynamically adjusts threshold values and evaluation criteria based on the importance and characteristics of different data types. Critical data types can have dynamically adjusted threshold values that reflect their importance, allowing the system to be more restrictive for critical types while more permissive for others, thus balancing reliability and productivity.
Solution Approach 2:
Different parameter settings and threshold values are applied to different data type groups. By changing evaluation parameters locally for each data type, the system can maintain high reliability for critical data while allowing overall performance improvement through updates that benefit other data types.
3Ease of operation
If simplistic performance comparison is used, then evaluation process is simple and fast, but validness of model update determination is insufficient
Solution Approach 1:
The evaluation process is segmented into multiple stages: initial overall performance evaluation, followed by detailed group-specific evaluations. This segmentation maintains operational simplicity at the high level while introducing precision through systematic breakdown into manageable evaluation components for each data type group.
Solution Approach 2:
The system performs partial evaluations first (overall performance) and then applies excessive action by conducting additional group-specific evaluations only when necessary. This approach maintains simplicity for clear cases while ensuring precision for borderline cases where model update determination is less obvious.
Data Source
AI summary
A model maintenance device according to an embodiment performs maintenance of a model for pattern recognition used in label estimation of target data for recognition. The model maintenance device includes a generating unit, an evaluating unit, and an updating determining unit. The generating unit generates a new model using learning data. The evaluating unit evaluates the performance of the new model using evaluation data classified into a first group, from among evaluation data classified into a plurality of groups, and calculates a first performance evaluation value; and evaluates performance of the new model using evaluation data classified into a second group, from among the evaluation data classified into a plurality of groups, and calculates a second performance evaluation value. Based on the first performance evaluation value and the second performance evaluation value, the updating determining unit determines whether or not the existing model should be updated with the new model.