N-Gram Index Segmentation for Document Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The N-Gram index system faces a trade-off between retrieval speed and index data size, where increasing the size of N improves search speed but increases index data size exponentially, and decreasing N increases the number of candidates, leading to longer search times, making it difficult to efficiently register and retrieve documents.
Innovation Solution
The method involves shifting character strings from document data to generate management Grams, determining their occurrence frequency, and storing them in separate regions based on low or high frequency, reducing the apparent number of Gram classes and optimizing registration and retrieval times by using integrated and general Gram post regions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the size of N is increased in the N-Gram index system, then retrieval speed is enhanced, but index data size increases exponentially
Solution Approach 1:
The patent segments the index data structure into two distinct parts: a compressed Gram information region that stores only unique Gram values with compact encoding, and a separate post region that stores the actual document position information. This segmentation allows the Gram classes to be stored efficiently without redundant data, while maintaining fast retrieval through the separate post lookup mechanism.
Solution Approach 2:
The patent changes the storage parameter of Gram data from storing complete character strings to storing compressed Gram values with reduced redundancy. By modifying how Gram information is encoded and stored (using parameter changes in the data structure), the index data size is reduced exponentially while maintaining the retrieval speed benefits of larger N values.
2Quantity of substance
If the size of N is decreased in the N-Gram index system, then the number of Gram classes decreases, but the number of candidates increases leading to longer search times
Solution Approach 1:
The patent extracts the position information from the Gram information and stores it in a separate post region. This extraction allows the Gram information region to focus solely on storing unique Gram values in compressed form, reducing the number of Gram classes stored. The separate post region efficiently handles the candidate matching, reducing search time by avoiding redundant comparisons in the Gram storage area.
3Adaptability or versatility
If the size of N is increased, then the number of kinds of indexes (Gram classes) increases, but this leads to larger index data size
Solution Approach 1:
The patent merges the storage of unique Gram values with the storage of position information by creating an integrated index structure where the Gram information region and post region work together. This merging allows the system to handle a large number of Gram classes (high adaptability) while keeping the total index data size manageable through the compressed storage of Gram values combined with efficient post region lookup.
Data Source
AI summary
A document management method includes shifting a character string of characters from document data and clipping it, determining that a management Gram obtained by the clipping is one of a first Gram of low frequency and a second Gram of high frequency, storing first post data in a first post region in association with a Gram value obtained by computing the character string of first Gram, the first post data having a set of a document identification (ID) indicating the document data including the first Gram and an intra-document offset indicating a character string position thereof, and storing second post data in a second post region in association with the character string of second Gram, the second post data having a set of a document identification (ID) indicating document data including the second Gram and an intra-document offset indicating a character string position thereof.