Indexing Unique Expressions via Hinted Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information retrieval systems face challenges in efficiently finding specific information from large datasets due to the inability to recognize unique expressions, such as organization names or project names, which are not registered as keywords, and the limitations of current indexing methods like n-gram and morphological analysis in handling variations and noise.
Innovation Solution
A system that creates indexes for information retrieval using a combination of first and second analysis methods, where hinting information is associated with unique expressions, allowing for efficient retrieval by switching between n-gram and morphological analysis based on content, effectively handling variations and noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If general keyword search is used to find information from accumulated documents, then search coverage is broad, but search precision deteriorates due to enormous data volume and inability to recognize unique expressions
Solution Approach 1:
The patent segments the text into two distinct parts: unique expressions (identified by hinting information) and ordinary text. Different indexing methods are applied to each segment - n-gram analysis for unique expressions and morphological analysis for ordinary text. This segmentation allows the system to handle each type of content appropriately, improving search precision without being overwhelmed by the total data volume.
Solution Approach 2:
The patent applies different indexing qualities to different parts of the text. For unique expressions marked with hinting information, n-gram analysis creates contiguous sequence indexes that preserve the exact wording. For other text, morphological analysis divides into meaningful units. This local differentiation of indexing quality enables precise retrieval of unique expressions while maintaining efficient processing of general text.
2Measurement precision
If n-gram analysis method is used to generate contiguous sequences for unique expressions, then retrieval accuracy for specific terms improves, but processing complexity increases
Solution Approach 1:
The patent performs preliminary action by associating hinting information with unique expressions before the indexing process. This pre-marking of unique expressions allows the system to quickly identify which parts of the text require n-gram analysis, avoiding the need to apply complex n-gram processing to the entire text corpus, thereby reducing overall processing complexity while maintaining high retrieval accuracy for specific terms.
3Productivity
If morphological analysis is used to divide text into meaningful units, then general search effectiveness improves, but handling of unique expressions with variations and noise deteriorates
Solution Approach 1:
The patent implements a dynamic indexing approach that switches between morphological analysis and n-gram analysis based on the presence of hinting information. For text without hinting markers, morphological analysis divides text into meaningful units for effective general search. When hinting information is detected, the system dynamically switches to n-gram analysis to preserve the exact contiguous sequence of the unique expression, ensuring reliable recognition even with variations and noise.
4Measurement precision
If combination search using keyword plus specific term is performed, then retrieval precision for unique expressions improves, but search operation complexity increases
Solution Approach 1:
The patent makes the search system self-service by automatically identifying unique expressions through hinting information and applying appropriate indexing methods without requiring user intervention. Users can perform simple keyword searches, and the system automatically handles the complexity of combining keywords with specific terms by recognizing hinting-marked unique expressions and retrieving them with high precision, thus maintaining operational simplicity while achieving high retrieval precision.
Data Source
AI summary
The present invention may be a system for creating indexes for information retrieval comprises a processor and a memory. The memory has program instructions embodied therewith. The program instructions are executable by the processor to cause the system to read a document having hinting information into a memory, where the hinting information is associated with each unique expression in an original document. The program instructions are further executable to create the indexes from the document, where a first analysis method for generating a contiguous sequence of items from a text in the document is used for creating the indexes for each sequence in the unique expression with which the hinting information is associated and a second analysis method for dividing the text into meaningful units is used for creating the indexes for each word in the text other than the unique expression.