Keyword Extraction Using Multi-Level Corpus Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword extraction methods, such as tf-idf, face challenges in accurately identifying keywords in documents when a term frequently appears across internal documents, leading to high inverse document frequency values and incorrect keyword selection, and are affected by the size of the corpus, which can result in inaccurate differentiation of term importance.
Innovation Solution
A system utilizing multiple types of corpora (general, enterprise-level, and employee-level) to generate multiple inverse document frequency values for each term, with normalization and preference given to smaller magnitude values to prevent 'garbage' terms and under-observed terms from being selected as keywords, and adjusting the function for inverse document frequency calculation based on corpus size to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single corpus is used for keyword extraction, then the calculation is simple, but the accuracy of term importance differentiation deteriorates
Solution Approach 1:
The patent divides the corpus into multiple hierarchical levels (corpus level, document level, term level) and uses multiple corpora (general corpus, enterprise corpus, employee corpus) to calculate inverse document frequency at different levels. This segmentation allows the system to capture term importance from multiple perspectives, resolving the contradiction between simplicity and accuracy by organizing complexity in a structured manner.
Solution Approach 2:
The patent extends the traditional single-dimension keyword extraction by adding multiple dimensions through hierarchical corpus levels. Instead of using a single corpus, the system calculates IDF across multiple corpus levels (general, enterprise, employee), creating a multi-dimensional view of term importance that improves differentiation accuracy while maintaining manageable complexity through systematic organization.
2Device complexity
If traditional tf-idf is used, then the method is simple, but the ability to capture reliable statistics for special terms deteriorates
Solution Approach 1:
The patent segments the statistics collection process into multiple corpus levels (general corpus for common terms, enterprise corpus for organizational terms, employee corpus for specialized terms). This segmentation enables reliable statistics capture for special terms by collecting data from appropriate corpus levels without requiring a completely complex new method, thus resolving the contradiction between simplicity and reliability.
Solution Approach 2:
The patent performs preliminary statistics collection across multiple corpus levels before keyword extraction. By pre-calculating inverse document frequency values at different corpus levels and storing them for later use, the system ensures reliable statistics are available for special terms while keeping the actual keyword extraction process simple and efficient.
3Measurement precision
If multiple corpora are used, then term importance differentiation improves, but the system complexity increases
Solution Approach 1:
The patent organizes multiple corpora into a hierarchical structure with clear levels (general corpus, enterprise corpus, employee corpus). Each level serves a specific purpose and contains terms with different importance characteristics. This segmentation allows the system to use multiple corpora for improved term importance differentiation while managing complexity through systematic organization and level-specific processing.
Solution Approach 2:
The patent applies different processing approaches to different corpus levels based on their local characteristics. General corpus terms receive different treatment compared to enterprise or employee corpus terms. This local quality approach allows the system to optimize for each corpus level's specific needs, improving overall term importance differentiation while avoiding unnecessary complexity in processing all corpora uniformly.
4Quantity of substance
If frequent internal terms are included in the corpus, then comprehensive coverage is achieved, but the accuracy of keyword selection deteriorates due to high IDF values
Solution Approach 1:
The patent segments the corpus into multiple levels, with frequent internal terms primarily appearing in enterprise and employee corpora rather than the general corpus. This segmentation allows comprehensive coverage to be achieved while preventing frequent terms from dominating the general corpus statistics, thus maintaining keyword selection accuracy through level-specific term distribution.
Solution Approach 2:
The patent adds corpus level as an additional dimension to term frequency analysis. Instead of a single corpus where frequent terms dominate, the system analyzes term frequency across multiple corpus levels (general, enterprise, employee). This multi-dimensional approach allows comprehensive coverage of all terms including frequent internal terms, while the hierarchical structure prevents these terms from receiving excessively high IDF values that would compromise keyword selection accuracy.
Data Source
AI summary
In one embodiment, a method includes: receiving data representing a plurality of corpora, each of the plurality of corpora including a set of documents; receiving data representing terms that appear in the corpora; for each one of the terms, determining a plurality of inverse document frequency values each associated with a respective one of the plurality of corpora; receiving data representing a subset of the terms that also appear in a document; for each term in the subset, determining a term frequency for the term in the document; and for each term in the subset, determining, an augmented term frequency-inverse document frequency value based on: (i) the term frequency, and (ii) the plurality of inverse document frequency values that were determined for the term in the subset.