Text Mining via Document Structure Label Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text mining technologies face challenges in efficiently extracting information from documents with structured contents, particularly when common expressions and formulaic section headings are present, leading to resource-intensive preprocessing steps.
Innovation Solution
A computer-implemented method and system that extracts frequent sequences from a set of documents, evaluates the confidence of these sequences as label expressions, and uses this confidence to extract keywords from target documents, thereby improving resource efficiency and reducing preprocessing workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional text mining methods are used on documents with structured contents, then information extraction can be performed, but resource consumption increases and preprocessing becomes more intensive
Solution Approach 1:
The system performs preliminary extraction of frequent sequences and their confidence evaluations on a corpus of documents before processing target documents. By pre-computing the frequent sequences and their confidence scores from the corpus, the system avoids reperforming these computationally intensive tasks on each individual target document, thus reducing overall resource consumption while maintaining text mining efficiency
Solution Approach 2:
The invention extracts only the essential frequent sequences and their confidence values from the document corpus, separating these key features from the rest of the document content. By taking out only the necessary information (frequent sequences with confidence > threshold) for subsequent keyword extraction, the system reduces the amount of data processing required while preserving the core functionality of text mining
2Measurement precision
If frequent sequence extraction is performed on all documents, then accurate label identification is achieved, but processing time increases
Solution Approach 1:
The system performs the time-consuming frequent sequence extraction and confidence evaluation in advance on a corpus of documents, creating a pre-computed reference set. When processing target documents, the system only needs to match against this pre-computed set rather than performing extraction from scratch, significantly reducing processing time while maintaining high accuracy in label expression identification
Solution Approach 2:
The system uses confidence scores as feedback to filter and prioritize frequent sequences. By evaluating the confidence of each frequent sequence and only considering those above a threshold, the system efficiently narrows down the candidate label expressions, reducing the search space and processing time while maintaining identification accuracy
Data Source
AI summary
Frequent sequences extracted from a set of documents according to a common rule are obtained. Based on comparing occurrence frequencies of various sequences, confidence of the first frequent sequence being a label expression representing a document part in a target document is evaluated. Keywords are extracted from the target document based on evaluation of the confidence.