Iterative Rule and Dictionary Creation for Text Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for creating analysis rules and dictionaries for extracting information from textual data require users to manually provide token sequences and words, which can be time-consuming and prone to oversight, especially in large datasets.
Innovation Solution
A computer-implemented method that employs an improvement cycle to recommend token sequences and dictionary words based on system-generated suggestions, allowing users to select and iteratively refine rules and dictionaries, assisting in the rapid identification and inclusion of relevant terms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If users manually provide token sequences and words for creating analysis rules, then the rules can be created with user control, but the process becomes time-consuming and prone to oversight
Solution Approach 1:
The system performs self-service by automatically generating candidate token sequences and dictionary words from the corpus using statistical analysis, eliminating the need for users to manually provide all rules and dictionaries. The system identifies patterns and suggests candidates based on frequency and co-occurrence analysis.
Solution Approach 2:
The system implements feedback by allowing users to review, select, and refine the automatically generated candidates. User selections and corrections feed back into the system to improve future suggestions, creating an iterative improvement cycle that enhances both speed and accuracy.
2Manufacturing precision
If users manually create dictionaries with several words for each rule, then the rules can be precisely targeted, but the process becomes complex and tedious
Solution Approach 1:
The system automatically generates candidate dictionary words by analyzing the corpus for frequently co-occurring terms and statistically significant patterns, eliminating the manual effort of creating dictionaries while maintaining precision through statistical validation.
Solution Approach 2:
The system performs preliminary analysis of the corpus to pre-generate candidate token sequences and dictionary words before the user creates rules. This preliminary action reduces the complexity of rule creation by providing ready-to-use candidates that users can selectively adopt.
3Loss of information
If the system processes large datasets to identify all relevant terms, then completeness of extraction improves, but the computational resources and time required increase significantly
Solution Approach 1:
The system applies partial action by generating a focused set of high-probability candidate token sequences and dictionary words based on statistical thresholds, rather than exhaustively processing all possible combinations in the corpus. This approach achieves sufficient completeness for practical purposes while maintaining high productivity.
Solution Approach 2:
The system changes parameters by using statistical frequency and co-occurrence metrics to prioritize candidate generation, focusing computational resources on the most promising terms and patterns. This parameter-based approach balances completeness and speed by targeting the most relevant portions of the corpus.
Data Source
AI summary
In an approach to rule and dictionary creation based on an improvement cycle, a computing device receives a corpus, wherein the corpus comprises textual data. The computing device stores, in a rule database, a rule based on a user input. The computing device stores, in a dictionary, one or more words based on the corpus. The computing device updates one or more of the rule database and the dictionary based on an improvement cycle comprising user selections from system-generated recommendations.