Encoded File Block Indexing for Faster Compressed Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing compression techniques, such as the LZ77 system, face inefficiencies when searching for character strings in combined files, as they require decompressing individual files and checking them against a search string, leading to slow search speeds even with indexing.
Innovation Solution
The approach involves encoding multiple files using a specific format, dividing the combined encoded data into fixed-length blocks, and creating a bitmap type index associated with each block to facilitate high-speed searches by indicating the presence of words within each block.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If files are individually compressed using LZ77 system with sliding window, then compression efficiency is improved, but search speed deteriorates because all files must be decompressed to perform searches
Solution Approach 1:
The patent divides the combined encoded data into multiple fixed-length blocks, creating a segmented structure that allows selective processing. Each block is independently indexed and can be searched without decompressing the entire file, resolving the contradiction between compression efficiency and search speed.
Solution Approach 2:
The patent creates bitmap indexes for each block during the encoding phase, performing preliminary indexing actions before search operations. This allows search queries to directly access pre-computed index structures rather than decompressing data during search, maintaining both compression efficiency and search speed.
2Adaptability or versatility
If index is created in units of individual files, then file-level search capability is improved, but search speed deteriorates because the index cannot efficiently handle combined file searches
Solution Approach 1:
The patent introduces a new dimensional organization by creating indexes at the block level rather than file level. This dimensional shift from file-based to block-based indexing enables efficient searches across combined files while maintaining the ability to handle individual file queries, as blocks can be selectively accessed regardless of file boundaries.
3Reliability
If all files are decompressed from the top to perform search, then complete search coverage is improved, but processing time deteriorates significantly
Solution Approach 1:
The patent extracts only the necessary blocks for search operations using bitmap indexes, rather than decompressing entire files. The bitmap index indicates which blocks contain target character strings, allowing the system to extract and process only relevant blocks, ensuring complete search coverage while minimizing processing time.
4Stability of the object's composition
If combined encoded data is searched without block division, then data integrity is maintained, but search efficiency deteriorates due to inability to perform selective decompression
Solution Approach 1:
The patent segments combined encoded data into fixed-length blocks while maintaining the overall data integrity through proper indexing. Each block is independently indexed with bitmap indexes that track character string presence, allowing selective decompression of specific blocks without compromising the integrity of the complete dataset.
Data Source
AI summary
A non-transitory computer-readable recording medium stores therein an encoding program that causes a computer to execute a process including: first creating a plurality of pieces of encoded data that are obtained by encoding a plurality of files by using a specific encoding format; second creating a plurality of encoded blocks that are obtained by dividing combined encoded data, the combined encoded data being obtained by combining the plurality of pieces of the encoded data into blocks with a fixed length; and third creating an index associated with each of the plurality of the encoded blocks.