Dual-Dictionary Text Compression for Unregistered Term Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression techniques using static dictionaries result in increased lengths of compression codes for words not registered in the dictionary, leading to a lower compression rate, especially for unique names and terms like personal names, places, and work names, which are not registered in the static dictionary.
Innovation Solution
An encoding and decompression program that utilizes both static and dynamic dictionaries, where unregistered character strings are stored in a dynamic dictionary and registered with a registration number, preventing the increase in data length of compression codes by converting them into compression codes based on the dynamic dictionary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a static dictionary is used for compression, then compression processing is simple and fast, but the compression rate deteriorates for words not registered in the dictionary
Solution Approach 1:
The patent divides the dictionary into two distinct segments: a static dictionary for common words and a dynamic dictionary for unique or less frequent words. This segmentation allows the system to handle different types of words with appropriate strategies, improving overall compression rate without overwhelming complexity
Solution Approach 2:
The patent introduces a dynamic dictionary that can be updated during compression processing based on the frequency of word appearances. This dynamic adaptation allows the system to improve compression rates for previously unseen words while maintaining the simplicity of the static dictionary for common words
2Productivity
If unique words are converted to character-level codes, then all words can be compressed, but the compression code length increases
Solution Approach 1:
The patent creates a copy mechanism where frequently occurring unique words are copied into the dynamic dictionary with their own compression codes. This allows subsequent occurrences to be compressed efficiently without falling back to character-level codes, thus maintaining shorter compression code lengths
3Productivity
If a dynamic dictionary is used alone, then compression rate improves for unique words, but compression processing becomes less efficient in the leading portion of data
Solution Approach 1:
The patent performs preliminary actions by pre-loading the static dictionary with common words before compression begins. This allows the system to immediately compress common words efficiently without needing to build the dynamic dictionary from scratch during processing
Solution Approach 2:
The patent merges the advantages of both static and dynamic dictionaries into a unified compression system. The static dictionary provides immediate compression for common words, while the dynamic dictionary adapts to capture unique words, achieving both speed and compression efficiency
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An information processing apparatus (100a) according to an embodiment determines whether a target character string is registered in a first dictionary, the target character string being a compression target contained in input data, outputs a compression code corresponding to the target character string when the target string is registered in the first dictionary, searches the target character string in first data when the target character string is not registered in the first dictionary, the first date accumulating character strings that are a part of the input data and have been determined to be not registered in the first dictionary, registers a matched character string in a second dictionary different from the first dictionary when the target character string is searched in the first data and outputs a compression code corresponding to a registration number of the target character string in the second dictionary.