Hardware Data Compression Using Common-Value Chunk Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware compression techniques are overly conservative for large regular data arrays, often limited to complex or less effective codes, and face complications with frequent data updates, such as page mapping and fragmentation.
Innovation Solution
The method involves sampling data chunks to extract common portions, storing only the remainder, and using hardware logic to achieve simple, fast, and random-access friendly compression/decompression, exploiting value locality in columnar stores for improved compression ratio and speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If existing hardware compression techniques are used for large regular data arrays, then device complexity is reduced, but compression ratio deteriorates and manufacturing precision is limited
Solution Approach 1:
The patent segments data into fixed-size chunks and processes them through hardware compression logic. Each chunk is divided into fields that are compressed independently using the common portion extraction technique, allowing the hardware device to maintain simplicity while achieving better compression ratios on regular data arrays.
Solution Approach 2:
The patent changes the parameter representation by extracting common portions (prefixes) from multiple data fields and storing them once. The remaining unique portions are stored separately, transforming the storage parameter from full field repetition to differential encoding, which improves compression ratio without increasing hardware complexity.
2Loss of substance
If software compression is used to achieve greater compression ratio, then loss of substance is reduced, but use of energy increases and productivity decreases
Solution Approach 1:
The patent replaces software-based compression algorithms with dedicated hardware compression logic implemented in ASIC or FPGA. This substitution of mechanical/computational system with specialized hardware circuitry achieves compression ratios approaching software levels while consuming significantly less power and providing faster compression speeds.
3Productivity
If hardware compression is used for frequent data updates, then productivity is improved, but device complexity increases due to page mapping and fragmentation management
Solution Approach 1:
The patent performs preliminary organization of data into fixed-size chunks with predetermined structures before compression. By pre-organizing data into regular patterns with consistent field layouts, the hardware compression logic can operate efficiently on updates without needing complex dynamic page mapping or fragmentation management mechanisms.
Data Source
AI summary
Methods and devices are provided for data compression. Data compression can include receiving a plurality of data chunks, sampling at least some of the plurality of data chunks extracting a common portion from a number of the plurality of data chunks based on the sampling, and storing a remainder of the plurality of data chunks in memory.