Hierarchical Text Indexing for Real-Time Neighborhood Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neighborhood search techniques for text data analysis are inefficient, particularly when dealing with large amounts of text data and requiring real-time analysis, often resulting in poor search accuracy and operational complexity, especially in parallel processing by multiple computer systems.
Innovation Solution
A method that hierarchically identifies word occurrences in text data, creates compressed indices for faster comparison, and calculates estimated and actual neighborhood occurrences using bit calculations to expedite top-k neighborhood searches, allowing for real-time processing and improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional neighborhood search techniques are used to analyze large amounts of text data, then search accuracy can be maintained, but processing speed becomes insufficient for real-time analysis requirements
Solution Approach 1:
The patent pre-processes text data to create an inverted index structure before actual search queries are executed. This inverted index organizes word occurrences by document, enabling rapid retrieval during search operations without re-scanning the entire text corpus, thus achieving real-time processing speeds while maintaining accuracy.
Solution Approach 2:
The patent divides the text processing task into hierarchical levels: first organizing words by document (upper hierarchy), then by word position within documents (lower hierarchy). This segmentation allows the system to process and search through large text corpora efficiently by handling smaller, organized units rather than processing the entire dataset at once.
2Productivity
If parallel processing by multiple computer systems is used to improve processing speed, then throughput increases, but operational complexity and search accuracy deteriorate
Solution Approach 1:
The patent designs a single-computer system that autonomously performs all necessary text processing operations using hierarchical indexing and bit calculation methods. The system self-optimizes by using estimated value calculations to determine which words require full neighborhood search analysis, eliminating the need for complex parallel processing coordination while maintaining high productivity.
3Measurement precision
If traditional neighborhood search methods are applied to large text datasets, then comprehensive analysis is achieved, but computational complexity and operational difficulty increase
Solution Approach 1:
The patent replaces traditional mechanical scanning of text documents with a data structure-based approach using inverted indices and bit calculations. Instead of sequentially checking each word occurrence in the entire text corpus, the system uses pre-organized index structures and bitwise operations to rapidly determine neighborhood occurrences, significantly reducing computational complexity while maintaining measurement precision.
4Measurement precision
If full neighborhood search is performed for all words to ensure accuracy, then search precision is maintained, but processing time increases significantly
Solution Approach 1:
The patent applies partial action by using estimated value calculations to determine which words require full neighborhood search analysis. Instead of performing complete searches for all words, the system calculates estimated occurrence values using hierarchical index comparisons and only performs full searches for words where the estimated value meets or exceeds a predetermined threshold, thus reducing processing time while maintaining precision for relevant results.
Data Source
AI summary
A method includes hierarchically identifying occurrences of some of the words in the set of sentences; creating a first index for each of some of the words based on the upper hierarchy of occurrences identified for each word; receiving input of a queried word; hierarchically identifying occurrences of the queried word in the set of sentences; creating a second index based on the upper hierarchy of occurrences identified for the queried word; comparing the first index and the second index to calculate an estimated value for the number of occurrences of a word in the neighborhood of the queried word; and calculating the actual value of the number of occurrences of a word in the neighborhood of the queried word based on an upper hierarchy and lower hierarchy of the occurrences on condition that the estimated value is equal to or greater than a predetermined number.