Word Shape-Assisted Big Data Search Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for identifying fields and values in unstructured digital documents are inefficient due to reliance on manual heuristics and computationally expensive word-to-word comparisons, especially when dealing with large databases and documents containing errors or variations in formatting.
Innovation Solution
The implementation of word shape-assisted searching algorithms, which map words to reduced representation word shapes, allowing for faster hypothesis elimination and verification using a hierarchical approach, and deployment of a trained classifier for partial matching in cases of incomplete data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If word-to-word comparisons are used for field detection, then accuracy is maintained, but processing speed deteriorates and computational cost increases
Solution Approach 1:
The patent segments the field detection process into two distinct phases: (1) a fast filtering phase using word shapes that compares only the structural pattern of words, and (2) a precise verification phase using actual word texts. This segmentation allows the system to quickly eliminate non-matching database entries using lightweight shape comparisons, then apply computationally intensive text comparisons only to the small subset of candidates that pass the shape filter, thereby resolving the contradiction between speed and accuracy.
Solution Approach 2:
The patent introduces word shapes as an intermediary representation between the original text and the comparison process. Word shapes serve as a simplified mediator that captures the essential structural characteristics of words (length, case pattern, digit presence) without containing the full textual information. This intermediary enables rapid preliminary filtering while preserving the ability to perform accurate text-based verification on selected candidates.
2Adaptability or versatility
If manual heuristics are used for field detection, then adaptability to various document formats is improved, but computational complexity increases
Solution Approach 1:
The patent transforms the field detection problem from text-based parameter matching to shape-based parameter matching. By changing the parameters from actual word texts to abstracted shape representations (capturing length, case patterns, digit positions), the system achieves comparable adaptability across different document formats while dramatically reducing computational complexity. The shape parameters are invariant to specific text content but sensitive to structural variations across formats.
Solution Approach 2:
The patent employs inexpensive, computationally lightweight word shape representations as disposable filtering objects. These shape representations require minimal computational resources to generate and compare, serving as a cheap first line of defense that eliminates the need for complex manual heuristics in most cases. Only when shapes fail to distinguish entries does the system resort to more expensive text-based verification.
3Reliability
If large databases are searched using conventional methods, then comprehensive results are achieved, but processing time increases
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing word shapes for all database entries before the actual search operation. During query processing, these pre-computed shapes enable rapid filtering of the large database without requiring full text comparisons. This preliminary preparation creates an efficient lookup structure that maintains result completeness by preserving all candidate matches while reducing the time required to search through large databases.
Solution Approach 2:
The patent extracts the essential identifying characteristics from full database entries by creating word shape representations. This extraction process removes unnecessary textual details while retaining the key structural features needed for discrimination. By taking out only the essential shape parameters and storing them separately, the system enables fast searching of large databases while maintaining the ability to retrieve complete information for matching entries.
Data Source
AI summary
Aspects of the disclosure provide for systems and methods for word shape-assisted searches in big data applications. The systems and methods of the disclosure enable operations that identify a mapping scheme in which words are represented via word shapes with same word shapes capable of representing different words. Operations further include forming hypotheses that prospectively associate words in a document with target entries in a database, and eliminainge at least some of the formed hypotheses based on mismatch between sets of word shapes corresponding to the words of the formed hypotheses and word shapes of various database entries.