Compressed File Indexing With Static-Dynamic Word Dictionaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file indexing techniques face a trade-off between reducing search noise and maintaining a small data size, with N-gram indexes either producing high noise or large data sizes, and current methods fail to effectively manage the indexing of less frequent words.
Innovation Solution
A computer-readable medium stores a compression program that generates an index by using a bit filter and word bitmap index, where higher frequency words are assigned static codes and lower frequency words are dynamically coded, reducing search noise while keeping the index data size small.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If an index in 1-gram format is used, then the data size of the index is kept small, but large search noise occurs
Solution Approach 1:
The patent segments words into character units and uses bit filters to represent character combinations. Instead of storing complete words or N-grams, it breaks down words into individual characters and uses bit patterns to represent valid character sequences, thereby reducing index size while maintaining search accuracy.
Solution Approach 2:
The patent applies different indexing strategies to different parts of the search process. Bit filters are used for quick character-level filtering, while the full word index is used for precise matching. This local differentiation allows efficient noise filtering without requiring a complete word-level index for all operations.
2Object-generated harmful factors
If an index in N-gram format is used with larger N, then search noise is reduced, but the data size of the index increases significantly
Solution Approach 1:
The patent introduces a new dimension for indexing by using bit filters that represent character combinations in a binary space. Instead of storing N-gram sequences linearly, it maps character combinations to bit patterns, creating a dimensional transformation that reduces storage requirements while preserving the ability to filter search noise effectively.
Solution Approach 2:
The patent changes the parameter representation from storing actual character sequences to storing bit patterns that encode character combinations. This parameter transformation allows the system to represent the same information more compactly, reducing index size while maintaining the ability to distinguish valid word structures from noise.
3Measurement precision
If a complete word index is created for all characters, then search accuracy is improved, but the data size becomes unmanageably large
Solution Approach 1:
The patent extracts only the essential character-level information needed for search filtering, rather than storing complete word data. By taking out just the character combination patterns and representing them as bit filters, it achieves accurate search filtering without the overhead of storing full word indexes for all possible character sequences.
Solution Approach 2:
The patent creates a simplified copy of the word structure at the character level. Instead of copying complete words into the index, it creates bit pattern representations that capture the essential structural information needed for search accuracy, providing a compact copy that suffices for filtering purposes.
Data Source
AI summary
A storing unit stores appearance information of the extracted word in correspondence with a specific record of the specific word in an index data when a specific word extracted from a target file is stored in a static dictionary that stores a plurality of words and a plurality of compressed codes corresponding respectively to the plurality of words, the appearance information including, existence information indicating whether the specific word exists in the target file or the appearance information indicating how many times the specific word has appeared in the target file. The storing unit stores adds a specific record of the specific word to the index data and stores the appearance information when the specific extracted word is not stored in the static dictionary and stored in a dynamic dictionary whose records are added in a course of compression.