Descriptor List Compression Using Fibonacci Delta Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtual tape library (VTL) systems, data de-duplication using partially ordered data sets results in large descriptor lists that consume substantial storage space and impact input/output operations, with conventional compression techniques offering minimal effectiveness due to their design for text files.
Innovation Solution
The system employs Fibonacci encoding for the first record, followed by calculating and encoding delta values between subsequent records, which allows for more efficient compression of descriptor lists by representing smaller values with shorter sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data de-duplication is performed using descriptor lists in VTL systems, then storage space is saved by eliminating duplicate data, but the descriptor lists grow to tens of megabytes consuming substantial storage space and impacting I/O operations
Solution Approach 1:
The descriptor list is segmented into fixed-size blocks (e.g., 4KB blocks). Each block contains a limited number of descriptors, and the list is organized as a sequence of blocks rather than a single large structure. This segmentation limits the growth impact on I/O operations and allows efficient block-level management.
Solution Approach 2:
The patent extracts only the essential identifying information from full data segments to create compact descriptors. Instead of storing or processing complete data references, the system uses extracted fingerprint-like identifiers that are sufficient for de-duplication purposes, significantly reducing descriptor size.
2Quantity of substance
If conventional compression techniques (LZW, GZIP) are applied to descriptor lists, then some compression is achieved, but the effect is minimal because these algorithms are designed for text files
Solution Approach 1:
The patent changes the parameter representation from variable-length descriptors to fixed-width binary encodings. Each descriptor is represented as a fixed number of bits (e.g., 32-bit or 64-bit integers), transforming the data structure into a format that exhibits regularity and predictability, which is highly compressible with standard algorithms.
Solution Approach 2:
Different compression strategies are applied to different parts of the descriptor list based on local characteristics. The patent uses dictionary-based compression for repetitive descriptor patterns and run-length encoding for sequences with repeated values, adapting the compression method to the local structure of the data.
Data Source
AI summary
A system and method for compression of partially ordered data sets is provided. A first record of the data set is compressed by encoding the record using a Fibonacci encoding technique. Thereafter, for each subsequent record N, the N−1st record is subtracted from the Nth record before encoding the result, thereby allowing each subsequent record to store the difference (or delta) from the previous record.