Dataset Relevance Scoring via Keyword TF-IDF Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face difficulties in selecting relevant datasets for data science models due to the lack of effective sorting options when dealing with numerous datasets, as existing methods do not adequately account for the relevance of datasets to specific models.
Innovation Solution
A computer-implemented method that extracts keywords from both data science models and datasets, calculates relative frequencies and weights based on source categories, and uses modified TF-IDF algorithms to determine aggregated relevant scores, thereby ordering datasets by their relevance to the models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If datasets are ordered by name or modification time, then the sorting is simple and fast, but the user cannot efficiently find relevant datasets when there are many datasets and the user does not remember the name
Solution Approach 1:
The system pre-calculates and stores keyword frequencies and TF-IDF scores for all datasets before the user needs to search. This preliminary processing enables fast relevance calculation when users search, avoiding time-consuming computations during the selection process itself.
Solution Approach 2:
The patent introduces keyword frequencies and TF-IDF scores as intermediary metrics that bridge between dataset content and model requirements. These intermediaries enable the system to quantify and compare dataset relevance without requiring users to manually inspect each dataset.
2Measurement precision
If the system calculates detailed relevance scores using keyword frequencies and TF-IDF algorithms, then the dataset ordering becomes more accurate and relevant to models, but the computational complexity increases
Solution Approach 1:
The relevance calculation is segmented into distinct components: keyword extraction, frequency calculation, TF-IDF computation, and final scoring. Each component processes specific aspects independently, making the overall complex system manageable and maintainable while achieving high measurement precision.
Solution Approach 2:
The system transforms raw dataset and model information into standardized parameters (keyword frequencies, TF-IDF scores, relevance weights) that can be systematically compared and processed. This parameter transformation enables precise relevance measurement through mathematical operations rather than complex qualitative analysis.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer system for measuring relevance of datasets to data science models. One or more servers implement steps: extract keywords in each data science model; determine first relative frequencies of the respective keywords in each data science model, for each source group in the data science models; extract keywords in each dataset; determine second relative frequencies of the respective keywords in each dataset, for each source group in the datasets; determine weights of the keywords; calculate first aggregated relevant scores of the respective keywords in each data science model, based on the first relative frequencies and the weights; calculate second aggregated relevant scores of the respective keywords in each dataset, based on the second relative frequencies and the weights. One or more servers calculate similarity between vectors of the first and second aggregated relevant scores, based on a similarity measure between vectors.