RNN Term Correlation Mining for Large-Scale Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning systems are inadequate for efficiently mining large-scale documents to discover statistically reliable correlations between terms, which is crucial for various text understanding applications such as information retrieval, web search, and intelligent chatbots.
Innovation Solution
The implementation of a deep learning system using a Recurrent Neural Network (RNN), specifically Long Short-Term Memory (LSTM), to train on large-scale documents, generate feature representations, and predict correlation factors between terms, enabling the inference of relevant terms based on input terms by leveraging co-occurrence and relative position information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional deep learning systems are used to mine large-scale documents for term correlations, then the system can process text data, but the computational efficiency is insufficient and the system cannot effectively discover statistically reliable correlations
Solution Approach 1:
The patent segments the term correlation mining process into distinct computational stages: document preprocessing, term extraction, co-occurrence counting, and statistical correlation calculation. This segmentation allows each stage to be optimized independently, improving overall computational efficiency while maintaining correlation reliability through systematic processing
Solution Approach 2:
The patent changes key parameters including using sliding window sizes of 500-1000 tokens for context extraction, applying threshold values for correlation significance (p<0.05), and adjusting minimum document frequency parameters. These parameter optimizations enable efficient processing of large-scale documents while ensuring statistically reliable term correlation discovery
2Loss of information
If the system mines correlations between all term pairs in large-scale documents, then comprehensive correlation information can be obtained, but the computational complexity and processing time increase significantly
Solution Approach 1:
The patent extracts only the most relevant term pairs for correlation analysis by filtering terms based on document frequency thresholds and using sliding window techniques to identify co-occurring terms within contextual proximity. This extraction approach reduces the search space from all possible term pairs to only those with potential meaningful correlations, significantly reducing processing time while preserving important correlation information
Solution Approach 2:
The patent applies partial action by computing correlations for a subset of term pairs that meet predefined criteria (minimum document frequency, co-occurrence threshold) rather than calculating all possible correlations. This selective approach maintains completeness of important correlation information while avoiding unnecessary computational overhead from analyzing all term combinations
3Quantity of substance
If the system processes large-scale documents to discover term correlations, then valuable insights can be obtained, but the memory requirements and computational resources increase
Solution Approach 1:
The patent segments large-scale document processing into manageable chunks using sliding windows of specific token sizes (500-1000 tokens). Each window is processed independently for term extraction and co-occurrence analysis, allowing the system to handle large document collections with limited memory by processing documents in segments rather than loading everything into memory simultaneously
Solution Approach 2:
The patent uses efficient data structures and algorithms that create compressed representations of term frequencies and co-occurrence matrices. Instead of storing complete document texts in memory, the system maintains compact copies of term statistics that can be updated incrementally, reducing memory requirements while enabling processing of large-scale document collections
Data Source
AI summary
Presented are relation inference methods and systems that use deep learning techniques for data mining documents to discover a relation between terms of interest in a given field covering a specific topic. For example, in the healthcare domain, various embodiments of the present disclosure provide for a relation inference system that mines large-scale medical documents in a free-text database to extract symptom and disease terms and generates relation information that aids in disease diagnosis. In embodiments, this is accomplished by training and using an RNN, such as an LSTM, a Gated Recurrent Unit (GRU), etc., that takes advantage of a term dictionary to examine co-occurrences of terms of interest within documents to discover correlations between the terms. The correlation may then be used to predict statistically most probable terms (e.g., a disease) related to a given search term (e.g., a symptom).