Prevalence-Volume Ratio Matrix for Document Relevance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods, such as Latent Semantic Indexing and TF-IDF, are inadequate for segmenting a corpus of documents to provide insights into term relevance within specific segments, limiting their ability to identify the most relevant terms or job titles associated with a particular category.
Innovation Solution
A computer-implemented method and system that ingest a corpus of documents, segment them based on a search operator, calculate prevalence-volume values, and generate a prevalence-volume ratio (PVR) matrix to associate tokens with relevance scores, enabling the identification of relevant terms and job titles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If TF-IDF techniques are applied to determine document relevance, then the relevance scoring between documents is improved, but the ability to segment the corpus and provide segment-based insights is lost
Solution Approach 1:
The patent applies segmentation by dividing the corpus into distinct segments based on document categories or topics. The system creates segment-specific term frequency matrices and calculates TF-IDF scores separately for each segment, enabling both precise relevance scoring within segments and the ability to analyze different corpus portions independently. This resolves the contradiction by maintaining TF-IDF precision while adding segmentation versatility.
2Adaptability or versatility
If the corpus is segmented to provide segment-based insights, then the ability to identify relevant terms within specific categories is improved, but the overall document relevance scoring capability is reduced
Solution Approach 1:
The patent implements multi-functionality by creating a system that can perform both segment-specific analysis and overall document relevance scoring. The architecture computes TF-IDF matrices at multiple levels: segment-level matrices for category-specific insights and corpus-level matrices for overall relevance. This allows the system to provide both specialized segment analysis and comprehensive document scoring, resolving the contradiction between adaptability and measurement precision.
3Measurement precision
If term frequency-weighted relevance is used to identify unique terms, then the relevance between documents with unique terms is improved, but the ability to identify segment-specific relevant terms is lost
Solution Approach 1:
The patent adds a segment dimension to the traditional TF-IDF approach. Instead of computing a single term frequency matrix for the entire corpus, the system creates multiple dimensioned matrices organized by segments or categories. Each segment has its own TF-IDF calculations, adding a categorical dimension that enables both unique term identification and segment-specific term analysis simultaneously, resolving the contradiction between term relevance precision and segment-specific identification capability.
Data Source
AI summary
Techniques for prevalence-volume based relevance are provided. Corresponding systems and methods may include ingesting a corpus of documents; receiving a search operator; segmenting the corpus of documents into (i) a first set of documents that matches the search operator, and (ii) a second set of documents that do not match the search operator; extracting a first and second token list of tokens; calculating a prevalence-volume value for tokens included in the first and second token lists; generating a prevalence-volume ratio (PVR) matrix that associates tokens included in the first and/or second token lists with a PVR value, wherein the PVR value for a particular token is a ratio between the prevalence-volume value of the particular token for the first set of documents and the prevalence-volume value of the particular token for the second set of documents; and associating the search operator with the generated PVR matrix.