Search Query Dropout Buckets for Continuous ML Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing models, particularly entity and relationship machine learning models, require significant manual annotation and resource consumption for training, and encounter accuracy issues when encountering new documents outside their domain, leading to decreased model performance.
Innovation Solution
A system and method for generating training data that allows for ongoing, substantially continuous, and/or real-time generation of training data by logging search queries, identifying unknown entities and relationships, and automatically annotating them into a dropout bucket, which is then formatted and used to re-train the model with minimal user interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manually annotated documents are used for training the model, then the model accuracy is improved, but the time and resources consumed increase considerably
Solution Approach 1:
The system performs preliminary action by automatically collecting and storing unannotated search queries in a dropout bucket before they are needed for training. This preparation phase captures unknown entities and relationships in advance, so when training is needed, the data is already organized and ready for annotation or automatic processing, eliminating the need for time-consuming manual annotation at the training stage
Solution Approach 2:
The system implements self-service by automatically identifying unknown entities and relationships in search queries, storing them in the dropout bucket, and managing the collection without human intervention. The entity and relationship model automatically processes incoming queries, determines what is unknown, and populates the dropout bucket autonomously, reducing reliance on manual annotation efforts
2Adaptability or versatility
If new documents outside the domain are used to re-train the model, then the model can handle new data, but the overall model accuracy and consistency decrease
Solution Approach 1:
The system extracts only the unknown entities and relationships from incoming search queries and stores them separately in the dropout bucket, rather than incorporating all new data directly into training. This selective extraction allows the model to accumulate new information without being immediately exposed to potentially harmful out-of-domain data, maintaining accuracy while building adaptability over time
Solution Approach 2:
The system performs preliminary action by collecting and isolating unknown entities and relationships in the dropout bucket before they are used for training. This pre-processing step allows for careful review, validation, or controlled annotation of the extracted data, ensuring that only quality verified information is used to update the model, thus preserving accuracy while enabling adaptability
3Manufacturing precision
If manually annotated documents are generated for resource constrained enterprises, then the training data quality is improved, but the resource consumption increases considerably
Solution Approach 1:
The system implements self-service by automatically collecting, identifying, and organizing unknown entities and relationships from search queries without requiring manual annotation resources. The dropout bucket autonomously accumulates training data in a structured format, enabling resource-constrained enterprises to obtain high-quality training data with minimal human intervention and resource expenditure
Solution Approach 2:
The system performs preliminary action by pre-collecting and organizing training data in the dropout bucket before it is needed. This advance preparation includes automatic identification of unknown entities and relationships, structuring the data appropriately, and making it ready for future use, thereby eliminating the need for resource-intensive manual annotation when training actually occurs
4Reliability
If the model is re-trained frequently with new annotated documents, then the model stays updated, but the time and computational resources increase
Solution Approach 1:
The system performs preliminary action by continuously collecting and organizing unknown entities and relationships in the dropout bucket as they arise from search queries. This ongoing preparation means that when re-training is decided upon, the training data is already organized and ready to use, significantly reducing the actual re-training time and computational overhead while keeping the model current
Data Source
AI summary
Disclosed here are methods and systems for generating a re-training set of data based on unknown entities. In an embodiment, a method includes logging a plurality of full search queries, generating a dropout bucket, determining whether each full search query of the plurality of full search queries includes an unknown entity and/or a known entity with an unknown relationship, and populating the dropout bucket with each full search query of the plurality of full search queries determined to include the unknown entity and/or the known entity with the unknown relationship. The method further includes after a pre-selected time interval, transmitting the dropout bucket to a computing device configured to generate annotated dropout buckets and in response to reception of an annotated dropout bucket, generating a formatted file readable by a machine learning training algorithm, and re-training a machine learning model based on the formatted file.