Machine Learning Model Training via Ranked Cluster Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning models trained for Optical Character Recognition (OCR) face challenges in accurately identifying entities from new data sets due to variations, requiring continuous updates and manual data variant addition, which is time-consuming and database-overloading.
Innovation Solution
A method involving creating clusters from datasets with entity and non-entity samples, determining ranks based on identification and mismatch probabilities, and retraining models using weighted clusters to improve accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple variants of data are added to the database to train machine learning models for entity identification, then the model's ability to handle new variants improves, but the database becomes overloaded and the process becomes time-consuming
Solution Approach 1:
The patent extracts only the most informative data samples by creating clusters from the dataset and selecting representative samples from each cluster. Instead of adding all possible data variants to the database, the system identifies and extracts key representative samples that capture the essential variations, thereby improving model adaptability without proportionally increasing database size.
Solution Approach 2:
The patent changes the parameter of data selection from exhaustive inclusion to ranked selection based on cluster representation. By determining ranks for clusters based on their informativeness and selecting top-ranked clusters, the system optimizes the balance between model adaptability and database size, avoiding the need to store all possible data variants.
2Measurement precision
If manual selection and addition of data variants is performed to train machine learning models, then training accuracy improves, but the process requires expert knowledge and consumes significant time
Solution Approach 1:
The patent implements self-service by enabling the system to automatically perform data selection and model training without requiring manual expert intervention. The clustering algorithm automatically identifies informative samples, determines their ranks, and selects representative data for training, replacing the time-consuming manual process with an automated system that achieves comparable or superior accuracy.
Solution Approach 2:
The patent substitutes the mechanical manual process of expert data selection with an automated computational system. Instead of relying on human experts to manually select and add data variants, the system uses clustering algorithms and rank determination to automatically identify and select informative samples, significantly reducing time consumption while maintaining or improving training accuracy.
3Ease of manufacture
If conventional machine learning models are trained on similar datasets, then training is straightforward, but the models cannot accurately identify entities from new datasets with variations
Solution Approach 1:
The patent segments the dataset into multiple clusters based on data characteristics and entity attributes. By dividing the data into distinct clusters and selecting representative samples from each, the system creates a more diverse and comprehensive training set that better represents variations in new data, thereby improving identification accuracy on unseen datasets while maintaining manageable training complexity.
Solution Approach 2:
The patent performs preliminary clustering and rank determination on the training dataset before actual model training. This preliminary action identifies and prepares the most informative data samples in advance, ensuring that the model is trained on high-quality, representative data that captures various variations. This preparation step improves the model's ability to generalize to new data while keeping the training process systematic and manageable.
Data Source
AI summary
A method and a device for creating and training machine learning models is disclosed. In an embodiment, a method for training a machine learning model for identifying entities from data includes creating a first plurality of clusters from a first plurality of data samples in a first dataset and a second plurality of clusters from a second plurality of data samples in a second dataset. The method further includes determining a rank for each of the first plurality of clusters and a rank for each of the second plurality of clusters. The method includes retraining the machine learning model using at least one of the first plurality of clusters weighted based on the rank determined for each of the first plurality of clusters and at least one of the second plurality of clusters weighted based on the rank determined for each of the second plurality of clusters.