Static-Dictionary OZIP Codec for Low-Latency Random Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression codecs, such as ZLIB and LZO, are inefficient for high-performance, low-latency applications with random data access due to their complexity and unsuitability for hardware implementation, leading to increased processing overhead and costs, as well as inadequate support for highly random access workloads.
Innovation Solution
OZIP, a data compression codec that uses direct token encoding to a static dictionary, enabling low-cost and high-performance hardware implementations with parallel processing capabilities, and supports random access through embedded block mapping, avoiding costly conditional branching and dynamic dictionary rebuilding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If common compression codecs (ZLIB, LZO) are used, then compression ratio is improved, but processing overhead and hardware complexity increase
Solution Approach 1:
The patent segments the compression process into distinct phases: dictionary construction from training data, tokenization of input data using the static dictionary, and encoding tokens with arithmetic coding. This segmentation allows each component to be optimized independently for hardware implementation, reducing overall system complexity while maintaining compression effectiveness.
Solution Approach 2:
The patent performs preliminary action by constructing a static dictionary during an offline training phase using representative training data. This pre-computed dictionary is then reused during the actual compression process, eliminating the need for dynamic dictionary construction during real-time compression operations, thereby reducing processing overhead and hardware complexity.
2Loss of substance
If common compression codecs (ZLIB, LZO) are used, then compression capability is improved, but latency increases
Solution Approach 1:
The static dictionary is constructed in advance during an offline training phase, so that during actual compression operations, the encoder can immediately tokenize input data without spending time on dynamic dictionary construction. This preliminary action significantly reduces processing latency for real-time compression tasks.
Solution Approach 2:
The patent extracts the time-consuming dictionary construction process from the real-time compression path and performs it separately during offline training. This separation allows the main compression pipeline to operate with minimal latency, using only the tokenization and arithmetic coding steps during active compression.
3Adaptability or versatility
If dynamic dictionary construction is used, then compression adaptability is improved, but hardware implementation cost increases
Solution Approach 1:
The dictionary is constructed in advance during an offline training phase using representative training data that captures the characteristics of the data to be compressed. This pre-computed static dictionary provides adaptability to different data types while keeping the hardware implementation simple, as no dynamic dictionary construction logic is needed during real-time compression.
Solution Approach 2:
The patent separates the adaptability function into two stages: offline dictionary construction (which provides adaptability to different data types) and online compression (which uses the pre-adapted dictionary). This segmentation allows adaptability to be achieved without requiring complex dynamic dictionary construction hardware during real-time operations.
4Device complexity
If sequential access design is used, then algorithm simplicity is improved, but random access capability deteriorates
Solution Approach 1:
The patent performs preliminary action by constructing a block index during the compression process that maps uncompressed block offsets to compressed data locations. This index is built in advance and stored with the compressed data, enabling efficient random access without requiring sequential processing during decompression operations.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
A method, apparatus, and system for OZIP, a data compression and decompression codec, is provided. OZIP utilizes a fixed size static dictionary, which may be generated from a random sampling of input data to be compressed. Compression by direct token encoding to the static dictionary streamlines the encoding and avoids expensive conditional branching, facilitating hardware implementation and high parallelism. By bounding token definition sizes and static dictionary sizes to hardware architecture constraints such as word size or processor cache size, hardware implementation can be made fast and cost effective. For example, decompression may be accelerated by using SIMD instruction processor extensions. A highly granular block mapping in optional stored metadata allows compressed data to be accessed quickly at random, bypassing the processing overhead of dynamic dictionaries. Thus, OZIP can support low latency random data access for highly random workloads, such as for OLTP systems.