Document Retrieval Training Data Collection via Grid-Based Feature Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current document processing systems face challenges in efficiently collecting and categorizing training data for machine learning, particularly in deriving relevant document vectors and calculating similarity scores, which affects the accuracy and efficiency of document retrieval and classification.
Innovation Solution
A system comprising a vector generation unit, feature quantity extraction unit, grid division unit, and training data extraction unit that derives feature vectors, performs dimensionality reduction, calculates cosine similarity, and classifies documents into partial regions to select and balance training data, enabling effective machine learning and document retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If cosine similarity is used to select training data within a specific range, then the relevance of training data is improved, but the coverage of diverse document types deteriorates
Solution Approach 1:
The patent divides the document selection process into two independent segmentation dimensions: (1) similarity-based segmentation that selects documents within a specific cosine similarity range to ensure relevance, and (2) category-based segmentation that ensures coverage across multiple document types. By segmenting the selection criteria into these two independent dimensions, the system achieves both high similarity precision and broad data diversity simultaneously.
2Productivity
If dimensionality reduction is applied to feature vectors, then the processing efficiency is improved, but the information loss increases
Solution Approach 1:
The patent applies parameter changes by using PCA (Principal Component Analysis) to transform the feature vector parameters. PCA identifies the principal components that capture the maximum variance in the data, allowing dimensionality reduction while preserving the most important information. This statistical parameter transformation enables efficient processing with minimized information loss.
3Manufacturing precision
If multiple feature quantities are extracted and classified, then the training data quality is improved, but the system complexity increases
Solution Approach 1:
The patent introduces a categorical dimension as an additional classification axis alongside similarity-based classification. Documents are classified not only by their similarity score to the target document but also by their category labels. This multi-dimensional classification approach improves training data quality by ensuring diverse category coverage while maintaining systematic organization through structured grid-based selection.
Data Source
AI summary
A vector generation unit derives a reference feature vector and a document feature vector. A feature quantity extraction unit performs a dimensionality reduction process on the reference feature vector and the document feature vector so as to set a dimensional value as a first feature quantity and derives a cosine similarity between the reference feature vector and the document feature vector as a second feature quantity. A grid division unit classifies documents into first partial regions obtained by dividing a feature quantity space of the first feature quantity, and classifies the documents into second partial regions obtained by dividing a range of the second feature quantity. A training data extraction unit selects, for each combination of a first partial region and a second partial region, a document classified in both the partial regions and sets documents selected with respect to all combinations as training data.