Dynamic Facet Dictionary Management for Text Mining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional text mining techniques require rebuilding the index to check if added words function well, which is inefficient and limits the dynamic management of facets and their values in the dictionary management process.
Innovation Solution
A system and method that integrates text mining with dynamic facet dictionary management, where words are extracted from documents, annotated, and selectively added as facets with values, allowing for real-time analysis and application of dictionaries to document collections without re-indexing, enabling efficient management and updating of facets and facet values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional text mining techniques are used with external editors to manage facets, then dictionary management is possible, but the process requires rebuilding the index to check if added words function well, which is inefficient
Solution Approach 1:
The system pre-processes and stores word-frequency information and document metadata in advance, creating a foundation that allows facet additions to be evaluated without complete index rebuilding. The preliminary structuring of data enables incremental updates rather than full re-indexing operations.
Solution Approach 2:
The index is divided into separate components: a static document metadata section and a dynamic word-frequency section. When new facets are added, only the relevant word-frequency segments need updating, not the entire index structure. This segmentation allows partial re-indexing instead of complete index rebuilding.
2Measurement precision
If the index is rebuilt to check added words, then accuracy of text mining is improved, but the process becomes slower and less dynamic
Solution Approach 1:
The system implements dynamic facet management where the dictionary structure can be modified in real-time without static re-indexing constraints. Facets and their values can be added, removed, or modified on-the-fly, with the system adapting the word-frequency calculations dynamically rather than requiring static index rebuilding for accuracy.
Solution Approach 2:
The text mining process maintains continuous operation while facets are being managed. Instead of stopping to rebuild the entire index, the system continuously updates word-frequency information incrementally, allowing accurate text mining to proceed without interruption while maintaining precision through ongoing rather than periodic index maintenance.
3Adaptability or versatility
If external editors are used for dictionary management, then facet construction is possible, but the integration with text mining is limited and requires separate processes
Solution Approach 1:
The system merges the dictionary editor and text mining engine into a single integrated platform. The facet editor directly interacts with the text mining components, allowing facets to be defined and immediately applied to text mining operations without export/import between separate systems. This unification maintains flexibility while reducing integration complexity.
Solution Approach 2:
The integrated system provides multi-functionality where the same software platform performs both dictionary/facet management and text mining operations. The facet editor can create, modify, and test facets within the same environment that executes text mining, eliminating the need for separate external tools and reducing system complexity while maintaining adaptability.
Data Source
AI summary
Embodiments are directed to a system, computer program product, and method for text mining, and dynamic facet and facet value management and application to a document collection. Two or more words from a first document collection are extracted, with the extracted words being associated with an applied annotation. At least one word is selected from the extracted words, designated as a facet, and a value is selectively added to the facet. An analysis of the added value is dynamically performed, and a dictionary with the annotation, facet, and values is constructed and the dictionary is applied to the document collection. A targeted list of documents is returned from the dictionary application to the document collection.