Semi-Automatic Word Space Generation for Noise Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating search terms and relationship information between words and named entities are inefficient, often resulting in large, noisy lists that require manual filtering and are time-consuming, especially for complex concepts like money laundering or rare events, and struggle to effectively capture relationships between named entities.
Innovation Solution
The proposed solution involves generating context-specific word spaces using semi-automatic methods that include manual thresholding and feedback loops to prune lists, and applying Latent Dirichlet Allocation (LDA) and Labeled LDA for relationship mining between named entities, allowing for context-specific relevance judgments and supervised learning to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated methods are used to generate search terms based on lexicons and word ontologies, then the process becomes faster and less manual labor is required, but the resulting lists become very large and noisy with words less related to the concept
Solution Approach 1:
The patent segments the term generation process into multiple stages: initial automated expansion using lexicons and ontologies, followed by iterative refinement through user feedback. This allows the system to first generate comprehensive lists quickly, then progressively filter and improve relevance through structured user interactions, resolving the contradiction between speed and reliability.
Solution Approach 2:
The system implements feedback loops where user judgments about term relevance are collected and used to retrain the automated selection algorithms. This feedback mechanism allows the system to learn from user preferences and improve the quality of generated lists over time, maintaining both automation benefits and high relevance.
2Ease of manufacture
If a single score threshold is set across all seed term expansions to filter terms, then the filtering process becomes simple and automated, but words more relevant than words that are not removed may be eliminated since not all seed words are equally related to a concept
Solution Approach 1:
The patent applies local quality by creating context-specific thresholds for each seed term rather than using a global threshold. The system analyzes the specific context and relevance characteristics of each seed term (e.g., 'crime' vs. 'investment' in money-laundering) and adapts the filtering criteria accordingly, allowing precise control over term selection for each conceptual domain.
Solution Approach 2:
The filtering thresholds are made dynamic rather than static. The system adjusts thresholds based on user feedback and contextual analysis, allowing the filtering criteria to evolve and adapt to different seed terms and their specific relevance patterns, thereby improving selection accuracy while maintaining automated operation.
3Manufacturing precision
If manual domain expert creation of search term lists is performed, then high precision and relevance are achieved, but the process becomes tedious, time consuming, error-prone and expensive
Solution Approach 1:
The system introduces an intermediary automated system that acts as a bridge between complete automation and full manual creation. The automated system generates initial lists and performs preliminary filtering, then presents refined results to domain experts for final validation. This intermediary approach captures expert knowledge efficiently while reducing the time and effort required compared to purely manual processes.
Solution Approach 2:
The system performs preliminary actions by automatically generating comprehensive term lists and pre-filtering them using algorithms before presenting them to domain experts. This preliminary processing reduces the workload on experts to mainly review and validate rather than create from scratch, significantly reducing time consumption while maintaining high accuracy through expert oversight.
4Reliability
If inspection of the entire list is performed to remove noise, then term relevance is improved, but the process requires O(n) time where n is the number of terms in the expanded word space
Solution Approach 1:
The system applies partial action by performing iterative filtering on subsets of the term list rather than inspecting the entire list at once. It focuses computational resources on the most promising or problematic portions of the list based on preliminary analysis, achieving effective noise removal without requiring linear time complexity for the entire dataset.
Solution Approach 2:
The system performs preliminary filtering and sorting before detailed inspection, using automated algorithms to identify and remove obvious noise candidates. This preliminary action reduces the size of the list that requires careful human or detailed computational inspection, thereby reducing the overall time required while maintaining list cleanliness.
Data Source
AI summary
For generating a word space, manual thresholding of word scores is used. Rather than requiring the user to select the threshold arbitrarily or review each word, the user is iteratively requested to indicate the relevance of a given word. Words with greater or lesser scores are labeled in the same way depending upon the response. For determining the relationship between named entities, Latent Dirichlet Allocation (LDA) is performed on text associated with the name entities rather than on an entire document. LDA for relationship mining may include context information and/or supervised learning.