Keyword Arrangement Feature Extraction for Document Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing similarity determination methods face challenges in reducing data volume without compromising accuracy, especially when handling large datasets, as they often delete features randomly or based on frequency, which can lead to loss of important information and reduced retrieval accuracy.
Innovation Solution
A similarity determination apparatus that selectively deletes features with low appearance counts, ensuring that features with higher counts are preserved, and strategically controls the deletion of features within specific ranges to maintain data integrity and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If features are deleted randomly or based on frequency to reduce data volume, then data volume is reduced, but accuracy of similarity determination deteriorates
Solution Approach 1:
The patent changes the parameter for feature selection from random or frequency-based deletion to deletion based on appearance count thresholds. By setting a threshold for the number of appearances, the system selectively removes only those features that appear fewer times than the threshold, thereby reducing data volume while preserving features that are more representative and important for accurate similarity determination.
2Measurement precision
If all features are retained to maintain accuracy, then accuracy of similarity determination is maintained, but data volume increases
Solution Approach 1:
The patent extracts and removes specific features from the dataset based on their appearance counts. By identifying and deleting features whose appearance counts fall below a predetermined threshold, the system extracts only the essential features needed for accurate similarity determination, thereby reducing overall data volume while maintaining processing accuracy.
3Quantity of substance
If features with low appearance counts are deleted to reduce data volume, then data volume is reduced, but important information may be lost
Solution Approach 1:
The patent changes the criterion for feature deletion from arbitrary or frequency-based to appearance count-based with a predetermined threshold. This parameter change ensures that only features with sufficiently low appearance counts are deleted, while features that appear frequently enough to be considered important are preserved, thus reducing data volume without losing critical information.
Data Source
AI summary
A determination apparatus has a feature extraction unit and a similarity determination unit. The feature extraction unit counts a number of appearances of each keyword included in a piece of document information and deletes any arrangement including a keyword having the number of appearances less than a threshold under a condition where a number of types of keyword arrangements included in a certain range of the piece of document information is equal to or greater than a certain number and extracts, as features, a plurality of keyword arrangements from the piece of document information. The similarity determination unit determines a similarity between the different pieces of document information by comparing the features extracted from pieces of document information different from each other.