Classification Model Label Sampling via Budget Probability

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing classification systems for consumer products face challenges in accurately labeling data records due to irrelevant information, high costs of labeling, and inefficiencies in reusing labels as item databases grow, leading to suboptimal search results and increased computing resources required for accuracy determination.

Innovation Solution

A networked computer system comprising a classification server and an evaluation server that assigns estimated labels to item records, determines selection probabilities based on a labeling budget, and generates a sampling list for accuracy measurement, allowing for the reuse of previous labels and optimal sampling strategies.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If a new test data set is sampled every time an evaluation is required, then the evaluation accuracy is maintained, but the labeling cost and computing resources increase significantly

Engineering Contradiction:

Improveevaluation accuracyVSAvoidlabeling cost and computing resources

Core Design Contradiction:

Measurement precisionVSLoss of energy

Solution Approach 1:

The system pre-samples test data sets and stores them in a cache memory before they are needed for evaluation. This preliminary action allows the system to reuse previously sampled data for multiple evaluations, avoiding the need to sample new data every time an evaluation is required, thus reducing labeling costs and computing resources while maintaining evaluation accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates and stores copies of test data sets in a cache memory structure. These cached copies can be reused across multiple evaluation operations, eliminating the need to repeatedly sample and label new test data, thereby significantly reducing the loss of energy in terms of labeling cost and computing resources.

Inventive Principle:

Principle #26Copying

2Quantity of substance

If the number of product records in the database increases, then the system capacity is improved, but the computing time and resources required to determine classification accuracy increase

Engineering Contradiction:

Improvenumber of product recordsVSAvoidcomputing time and resources

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The system segments the large product database into manageable portions and uses probabilistic sampling to select representative subsets for evaluation. By dividing the full dataset into smaller sampled portions, the system can determine classification accuracy without processing the entire large database, thus reducing computing time and resources while maintaining system capacity to handle large numbers of product records.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs partial action by evaluating only a sampled subset of the product records rather than the entire database. This partial evaluation approach provides sufficient accuracy for determining classification performance without the computational burden of processing all records, effectively managing the trade-off between database size and computing resources.

Inventive Principle:

Principle #16Partial or excessive action

3Ease of manufacture

If labeled data is randomly divided between train and test sets, then the process is simple, but the evaluation may not be optimal or representative

Engineering Contradiction:

Improvedata splitting simplicityVSAvoidevaluation representativeness

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The system changes the parameters of the sampling process by using probabilistic sampling with carefully selected sample sizes and stratification methods. Instead of simple random division, the system employs controlled sampling parameters that ensure the test set is representative of the overall data distribution, thereby improving evaluation precision while maintaining practical simplicity in the data splitting process.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11216855B2Server computer and networked computer system for evaluating, storing, and managing labels for classification model evaluation and training

Publication Date: 2022.01.04 WALMART APOLLO LLC

AI summary

A system for use in monitoring an operation of a classification model in generating estimated labels for item records is described herein. The system receives a current labeling budget value including a number of trusted labels available for use in a labeling operation, determines a current selection probability for each item record included in an item list as a function of the current labeling budget value, selects a plurality of item records from the item list as a function of each corresponding current selection probability, and generates a sampling list including the selected item records. The system determines a risk measurement value associated with the classification model indicating an accuracy of the estimated labels as compared to trusted labels associated with the item records.