Encoding Apparatus Using Intermediate Codes for Fast Document Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document search processes are inefficient due to the need for decompression and the lack of commonality in compression and search processing, resulting in the inability to utilize total results generated during compression.
Innovation Solution
A computer-readable medium stores an encoding program that encodes documents using a static word dictionary and intermediate code table, allowing for frequency counting and optimal encoding, enabling search processing without full decompression and utilizing total results from compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If documents are compressed using traditional algorithms (ZIP/LZ77 or per-document word counting), then compression efficiency is improved, but search processing requires full decompression and cannot utilize compression results
Solution Approach 1:
The patent segments the compression process into two independent stages: (1) lexical analysis stage that generates word frequency information and intermediate codes, and (2) compression stage that encodes documents using these intermediate codes. This segmentation allows search operations to utilize the pre-computed lexical analysis results without requiring full decompression, thus resolving the contradiction between compression efficiency and search speed.
Solution Approach 2:
The patent performs preliminary lexical analysis and word frequency counting during the compression phase, storing these results as reusable metadata. When search operations are needed later, the system can directly query these pre-computed results without re-analyzing the document content, enabling fast search over compressed documents and eliminating the need for full decompression.
2Adaptability or versatility
If per-document word dictionaries are used for compression, then compression is adapted to each document, but total results cannot be shared across multiple documents
Solution Approach 1:
The patent merges the lexical analysis results from multiple documents into a unified global word frequency table. Each document is analyzed using the same standardized dictionary and encoding rules, allowing the system to aggregate word frequencies across all documents. This merging enables efficient multi-document search operations while maintaining document-specific compression through the use of intermediate codes.
Solution Approach 2:
The patent creates a universal intermediate code representation that serves multiple functions: it enables both efficient compression of individual documents and fast search operations across multiple documents. The standardized lexical analysis framework and global word frequency table provide a universal basis that works for any document in the collection, eliminating the need for separate per-document processing during search operations.
3Loss of energy
If sliding window longest-match compression is used, then compression ratio is improved, but word-level search capability is lost
Solution Approach 1:
The patent introduces intermediate codes as a mediator between the original document text and the compressed representation. During compression, documents are first converted to intermediate codes based on word boundaries and frequency information, then these intermediate codes are compressed using efficient encoding. This intermediary representation preserves word-level structure information while enabling high compression ratios, and allows search operations to work directly on the compressed intermediate code representation.
Data Source
AI summary
An information processing device encodes a plurality of documents into a first plurality of encoded documents, respectively based on first encoding information in which a plurality of words and a plurality of first codes of a first code group are associated, words included in the plurality of documents and included in the first encoding information being encoded in the plurality of encoded documents. The information processing device performs frequency counts for each of a plurality of codes in the first encoded documents encoded in the first encoding. The information processing device encodes the plurality of first encoded documents into a plurality of second encoded documents respectively, utilizing a result of the frequency counts.