Compressed File Search System Using Frequency-Based Huffman Tree

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing file search technologies in compressed archives face inefficiencies due to increased time for opening processes as the number of files searched increases, with 20-30% of file processing time dedicated to opening, leading to reduced search speed and storage fragmentation, especially in large-scale dictionary searches.

Innovation Solution

Implementing a system that sorts and combines compressed files based on access frequency, prioritizing files with high access rates for caching to reduce search time and optimize storage usage, and using a common Huffman tree for efficient file management across multiple servers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If files are compressed using different compression parameters, then compression efficiency is improved, but the time necessary for opening processes increases

Engineering Contradiction:

Improvecompression efficiencyVSAvoidopening process time

Core Design Contradiction:

Loss of substanceVSLoss of time

Solution Approach 1:

The patent pre-calculates and stores compressed file data in advance, organizing it by access frequency before actual search operations. This preliminary organization eliminates the need for real-time compression parameter calculations during opening processes, resolving the contradiction between compression efficiency and opening time.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts file organization based on access frequency, moving frequently accessed files to positions that require minimal opening processes. This dynamic reorganization optimizes the balance between maintaining compression efficiency and reducing opening time based on actual usage patterns.

Inventive Principle:

Principle #15Dynamics

2Quantity of substance

If the number of files to be searched increases, then search coverage is improved, but the percentage of files requiring opening processes drastically increases

Engineering Contradiction:

Improvesearch coverageVSAvoidsearch speed

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent segments the file system into frequency-based groups, organizing files according to their access frequency rather than treating them uniformly. This segmentation allows the system to efficiently handle large numbers of files by focusing opening processes only on relevant segments, maintaining search coverage while preserving search speed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the organizational parameter from uniform file structure to frequency-based hierarchical structure. By reorganizing files based on access frequency parameters, the system can scale to handle increasing numbers of files without proportionally increasing opening process overhead, thus maintaining productivity.

Inventive Principle:

Principle #35Parameter changes

3Manufacturing precision

If file management is executed for each sector, then storage precision is improved, but fragmentation of disc area occurs and storage size increases

Engineering Contradiction:

Improvestorage precisionVSAvoidstorage size

Core Design Contradiction:

Manufacturing precisionVSVolume of stationary object

Solution Approach 1:

The patent merges file management operations by organizing multiple files into frequency-based groups that can be managed collectively rather than individually per sector. This merging reduces disc fragmentation while maintaining precise control over file locations, thereby reducing overall storage size requirements.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system creates an organizational copy or index based on access frequency that directs file access without requiring physical replication of files across sectors. This virtual copying mechanism maintains storage precision while eliminating the need for redundant physical storage, reducing total storage size.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS8037035B2Apparatus for searching and managing compressed files

Publication Date: 2011.10.11 FUJITSU LTD

AI summary

A computer-readable, non-transitory medium stores a program that manages compressed file groups on a plurality of slave servers. The file groups include compressed files that are to be searched and have character strings. Each of the compressed file groups is expanded, using a Huffman tree that was used for compressing the compressed file group. A common compression parameter is generated based on appearance frequency, by summing, for each character, the appearance frequency in each of the compressed file groups. The expanded files are recompressed using the common Huffman tree such that sums of the access frequencies of the compressed files that are origins of the recompressed files are substantially equivalent among various slave servers. New archives including the re-compressed files are transmitted to the respective slave servers.