Compressed Search Indexing With Word Bitmap Noise Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search index technologies face a trade-off between reducing search noise and maintaining a small data size, with N-gram formats either leading to large data sizes or high search noise due to their inability to accurately record all relevant word sequences.
Innovation Solution
A compression program and search program that utilize a bit filter with static and dynamic dictionaries to efficiently store and retrieve words, assigning variable-length codes to high-frequency words and fixed-length codes to low-frequency words, and recording existence or frequency information in a word bitmap index to minimize search noise while keeping the index size small.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If an index in 1-gram format is used to keep the data size small, then the data size of the index is reduced, but large search noise occurs because individual characters are recorded separately
Solution Approach 1:
The patent segments the index into two distinct parts: a first storage region for high-frequency words (reducing noise) and a second storage region for N-gram sequences (capturing character combinations). This segmentation allows the system to handle different types of search terms optimally without compromising overall search accuracy or index size.
Solution Approach 2:
The patent applies different indexing strategies to different parts of the search space: high-frequency words are indexed by individual characters (1-gram) while lower-frequency terms are indexed by N-gram sequences. This local quality approach ensures that each part of the index is optimized for its specific purpose, balancing data size and search accuracy.
2Reliability
If an index in N-gram format is used to reduce search noise, then search accuracy is improved, but the data size of the index increases significantly
Solution Approach 1:
The patent applies N-gram indexing selectively only to lower-frequency words rather than all search terms. This partial action approach reduces the overall data size of the index while still capturing the character combination patterns needed to reduce search noise for appropriate search terms.
Solution Approach 2:
The patent changes the indexing parameter (from 1-gram to N-gram) based on the frequency of words. High-frequency words use 1-gram indexing while lower-frequency words use N-gram indexing, allowing the system to adapt the index structure to the specific requirements of different search terms and optimize the balance between data size and search accuracy.
3Quantity of substance
If variable-length codes are assigned to high-frequency words and fixed-length codes to low-frequency words, then compression efficiency is improved, but the complexity of code management increases
Solution Approach 1:
The patent performs preliminary classification of words by frequency before assigning codes. High-frequency words are identified and assigned variable-length codes in advance, while lower-frequency words are assigned fixed-length codes. This preliminary action simplifies the overall code management process by establishing a clear, pre-determined coding strategy.
Solution Approach 2:
The patent changes the code length parameter based on word frequency. Variable-length codes are used for high-frequency words to achieve better compression, while fixed-length codes are used for lower-frequency words to simplify management. This parameter change approach allows the system to optimize compression efficiency without excessively complicating code management.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A storing unit (114) stores appearance information of the extracted word in correspondence with a specific record of the specific word in an index data when a specific word extracted from a target file is stored in a static dictionary that stores a plurality of words and a plurality of compressed codes corresponding respectively to the plurality of words, the appearance information including, existence information indicating whether the specific word exists in the target file or the appearance information indicating how many times the specific word has appeared in the target file. The storing unit (114) stores adds a specific record of the specific word to the index data and stores the appearance information when the specific extracted word is not stored in the static dictionary and stored in a dynamic dictionary whose records are added in a course of compression.