Dataset Generation for Semantic Matching Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating negative samples in training datasets for semantic similarity matching models are inaccurate due to random selection of texts with different semantics, leading to false samples and simple data, which affects the accuracy and robustness of the matching model.
Innovation Solution
A method that involves obtaining text blocks with similar semantics, selecting a first text set meeting a preset similarity matching condition, generating a negative sample set from texts not belonging to the same text block as the target text, and a positive sample set from the target text block, to create a dataset for training a matching model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If random selection of texts with different semantics is used to generate negative samples, then the process is simple and fast, but the accuracy and robustness of the matching model deteriorates due to false samples
Solution Approach 1:
The patent segments the text corpus into multiple text blocks based on semantic similarity. Each text block contains texts with similar semantics, and negative samples are generated by selecting texts from different text blocks. This segmentation allows the system to maintain high-speed sample generation while improving accuracy, as texts from different blocks are guaranteed to have different semantics, eliminating false negative samples.
Solution Approach 2:
The patent performs preliminary action by pre-grouping texts into text blocks based on semantic similarity before generating negative samples. This pre-processing step organizes the data structure so that negative sample generation becomes a simple matter of selecting from different blocks, achieving both speed and accuracy without requiring complex real-time semantic analysis during sample generation.
2Ease of manufacture
If random selection of negative samples is used, then the data generation process is simple, but the quality and relevance of training data deteriorates
Solution Approach 1:
The patent divides the training data into segmented text blocks where each block contains texts with similar semantics. This segmentation enables simple random selection within blocks while ensuring that negative samples come from semantically different blocks, thereby maintaining high data quality without complicating the generation process.
Solution Approach 2:
The patent applies local quality by ensuring that within each text block, texts have similar semantics (high quality locally), while negative samples are selected from blocks with different semantic characteristics. This creates a structured approach where simplicity is maintained locally within blocks while global diversity ensures data quality for training.
Data Source
AI summary
The present disclosure discloses a method and apparatus for processing a dataset. The method includes: obtaining a first text set meeting a preset similarity matching condition with a target text from multiple text blocks provided by a target user; obtaining a second text set from the first text set, in which each text in the second text set does not belong to a same text block as the target text; generating a negative sample set of the target text based on content of a candidate text block to which each text in the second text set belongs; generating a positive sample set of the target text based on content of a target text block to which the target text belongs; and generating a dataset of the target user based on the negative sample set and the positive sample set, and training a matching model based on the dataset.