ANS Frequency Table Hardware With On-the-Fly Overrun Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression systems face inefficiencies due to the need to sort symbol frequency tables, which leads to high latency and potential overrun errors, especially when dealing with large datasets and near-zero symbol values, causing bottlenecks in normalization and encoding processes.
Innovation Solution
A parallel data compression system that normalizes symbol occurrences and corrects overrun errors on-the-fly during frequency table creation, allowing normalization, overrun compensation, and encoding table generation to proceed simultaneously, reducing latency and improving pipeline efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If frequency table sorting is performed to rank symbols by occurrence, then encoding efficiency is improved, but processing time and latency increase significantly
Solution Approach 1:
Instead of sorting the frequency table to improve encoding efficiency, the patent inverts the approach by using the raw unsorted frequency counts directly in the encoding process. The ASK encoder processes symbols in the order they appear in the input data, using their frequency counts without requiring any sorting operation, thereby eliminating the time-consuming sorting step while maintaining encoding effectiveness.
Solution Approach 2:
The patent performs frequency counting in advance during the first pass through the input data, storing the counts in a frequency table. This preliminary action allows the encoding to proceed immediately in the second pass without waiting for any sorting operations, thus reducing latency while preserving the ability to encode based on symbol frequencies.
2Loss of substance
If traditional two-stage encoding with run-length and entropy coding is used, then compression ratio is improved, but pipeline efficiency and throughput decrease due to sequential processing
Solution Approach 1:
The patent merges the frequency counting and encoding operations into a single unified ASK encoding process. Instead of performing run-length encoding followed by entropy coding in separate stages, the ASK encoder directly encodes symbols using their frequency counts, combining what were previously separate processing stages into one efficient operation that improves throughput while maintaining compression effectiveness.
Solution Approach 2:
The patent enables continuous processing by allowing the encoder to operate on symbols as they are encountered in the input stream, without waiting for complete preprocessing or sorting operations. The frequency table is built and used concurrently, maintaining a continuous flow of useful action throughout the encoding process, which eliminates pipeline stalls and improves overall throughput.
3Measurement precision
If frequency table is populated during compression and transmitted with compressed data, then decoding accuracy is improved, but bandwidth consumption increases due to overhead
Solution Approach 1:
The patent changes the representation parameters of the frequency information from traditional explicit frequency tables to compact ASK encoding parameters. Instead of transmitting full frequency counts for all possible symbols, the ASK encoder transmits only the necessary encoding parameters that can be used to reconstruct the encoding process at the decoder, significantly reducing the amount of overhead data while maintaining decoding accuracy.
4Manufacturing precision
If normalization is performed on occurrence values, then encoding precision is improved, but overrun errors occur with near-zero symbol values
Solution Approach 1:
The patent applies preliminary anti-action by using bias correction techniques before normalization to prevent overrun errors. The encoder adds appropriate biases to frequency counts of symbols with low occurrence values before performing normalization, which prevents these symbols from being rounded down to zero and causing overrun conditions during encoding, thus maintaining both precision and reliability.
Data Source
AI summary
A lossless data compressor prevents normalization overruns on-the-fly as symbol occurrence counts are rounded to generate symbol frequencies, allowing an encoding table generator to generate encoding table entries without waiting for the symbol frequency table to finish filling. Rounding errors are accumulated as symbols are normalized and compensated for by reducing a symbol frequency when the symbol frequency is at least 2 and the accumulated errors have exceeded a threshold. The symbol frequency is also reduced when the number of remaining states in the encoding table is insufficient for a number of remaining unprocessed symbols and states for a current encoding table entry. Since error compensation occurs as symbols are being normalized, encoding table generation is not forced to wait for all symbols in the block to be processed, reducing latency. Three pipeline stages can operate on three input blocks: symbol counting, normalization/error compensation/encoding table generation, and data encoding.