Template-Based Document Compression for Fast Random Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression schemes are inadequate for systems requiring high-performance random access to individual documents, as they either concatenate and compress all documents together, leading to inefficient storage and costly extraction of metadata records, especially in environments like Atmos® where metadata records have similar structures but differ slightly.
Innovation Solution
The Template Based Compression method constructs a template document representing the maximum common content of a set of similar objects, compresses it, and then subtracts the compressed template from the concatenated compressed result for each object, storing only the compressed difference, which significantly reduces storage capacity while allowing quick access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If all documents are concatenated and compressed together, then compression ratio is improved, but random access to individual documents becomes inefficient and metadata extraction becomes costly
Solution Approach 1:
The patent divides the compressed data into separate segments, each containing a document and its metadata, while sharing a common compressed template. This segmentation allows individual documents to be accessed independently without decompressing the entire dataset, resolving the contradiction between achieving high compression ratios and maintaining efficient random access capability.
2Ease of operation
If traditional compression schemes are used for metadata records, then individual record access is maintained, but storage capacity consumption is high
Solution Approach 1:
The patent merges multiple similar metadata records into a single compressed template that represents the common structure and content. Individual records are then reconstructed by combining the compressed template with their specific differences, achieving both high storage efficiency and maintained individual record accessibility.
3Quantity of substance
If a template representing common content is used for multiple documents, then storage capacity is reduced, but system complexity increases
Solution Approach 1:
The patent performs preliminary identification of the common template structure before compression. By pre-defining the template that represents the maximum common portion of content across multiple documents, the system simplifies the compression process and reduces storage requirements without introducing excessive complexity during the compression and decompression operations.
Data Source
AI summary
Example embodiments of the present invention effectively manage a large set of records such that each can be quickly accessed while still reducing the system capacity used for storing the records by taking into account specifics of the record structure. A template document is constructed for a large set of similar documents, such that it represents the maximum common portion of content in the document set. The template is compressed and stored. Every document in the set is then concatenated individually to the uncompressed template and the concatenated result is compressed. The compressed template is then subtracted from the combined compressed result. The result of this subtraction is stored in the data store for each document. Effectively, only the compressed difference between each document and the template is stored, which reduces significantly the amount of capacity necessary for storing the document set (e.g., by a factor of 5 or 10).