Keyword Weighting for MapReduce Document Association
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in effectively integrating and associating unstructured documents with structured data on Hadoop distributed file systems, particularly in identifying relevant structured data for unstructured documents, leading to inefficient data processing and analysis.
Innovation Solution
A processor extracts and weights keywords from structured and unstructured documents, compares them using a predetermined threshold, and removes unstructured documents from processing lists based on similarity scores, optimizing data integration and processing on the MapReduce platform.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If keyword extraction and comparison is performed for all unstructured documents to identify relevant structured data, then data integration accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The patent segments the large set of unstructured documents into smaller batches for parallel processing using MapReduce framework. The map phase extracts keywords from individual documents independently, while the reduce phase aggregates and compares keywords across documents. This segmentation enables accurate keyword comparison to improve data integration accuracy while distributing computational load to reduce processing time.
Solution Approach 2:
The patent performs preliminary keyword extraction from all unstructured documents in the map phase before the comparison operation. By pre-extracting and storing keywords with their weights during the map phase, the system prepares data in advance for the reduce phase comparison, thereby improving integration accuracy while optimizing the overall processing timeline through parallel preparation.
2Loss of information
If all unstructured documents are processed to ensure complete data integration, then data completeness is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent extracts only the essential keywords from unstructured documents rather than processing the entire document content. By identifying and extracting key terms with associated weights during the map phase, the system maintains data completeness for integration purposes while dramatically reducing the amount of data that needs to be compared and processed in the reduce phase, thereby improving processing efficiency.
Solution Approach 2:
The patent transforms unstructured document data into a standardized keyword-parameter format with weighted importance values. This parameter transformation during the map phase converts varied document formats into a uniform structure that can be efficiently compared and aggregated in the reduce phase, ensuring complete data integration while improving processing efficiency through standardized operations.
3Measurement precision
If keyword weights are calculated and compared for all documents, then association accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent introduces keyword weights as an intermediary parameter that mediates between raw document content and association accuracy. During the map phase, keywords are extracted and assigned weights based on their importance. In the reduce phase, these weighted keywords serve as intermediaries for comparison operations, enabling accurate association between structured and unstructured data while simplifying the computational process compared to analyzing full document content.
4Speed
If parallel processing is implemented across multiple nodes, then processing speed is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal MapReduce framework that can process any unstructured document type through the same map and reduce operations. The map function universally extracts keywords from any document format, and the reduce function universally compares and aggregates these keywords across all documents. This multi-functional approach enables parallel processing across multiple nodes to improve speed while maintaining consistent, manageable system complexity through standardized operations.
Data Source
AI summary
In an approach for integrating documents a processor extracts a first set of keywords from at least one structured document. A processor generates a first batch of keywords from the first set of keywords, wherein each keyword in the first batch of keywords includes a weight. A processor extracts a second set of keywords from at least one unstructured document. A processor compares the first batch of keywords to the second set of keywords. A processor determines that the at least one unstructured document matches, based on a predetermined threshold, the at least one structured document, based on the comparison of the first batch of keywords to the second set of keywords. A processor removes the at least one unstructured document from a list of unstructured documents which are to be processed.