Data Compression Analyzer Using CV*NZ Compressibility Decision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods face challenges in efficiently determining whether to compress data, especially in hardware implementations, due to the complexity of entropy-based compression ratio predictions and the need for logarithmic operators, which can be data-dependent and computationally intensive.
Innovation Solution
A neural network-based system that uses a Concentration Value (CV) and Number of Zeros (NZ) product to quickly decide on data compressibility by comparing the product to a threshold, eliminating the need for entropy calculations and logarithmic operators, and optimizing the threshold using a Receiver Operating Characteristic (ROC) curve for improved decision-making.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If entropy-based compression ratio prediction is used, then measurement precision of compression ratio is improved, but device complexity increases due to need for logarithmic operators
Solution Approach 1:
The patent replaces the mathematical entropy calculation mechanism (requiring logarithmic operators) with a neural network-based prediction mechanism. The neural network is trained offline to learn compression patterns, then deployed as a lightweight model that predicts compressibility scores without requiring complex runtime calculations, thus substituting a mathematically intensive system with a computationally efficient one.
Solution Approach 2:
The patent performs preliminary training of the neural network model offline using entropy-based compression ratio data. During this preliminary phase, the network learns to map input data characteristics to compression outcomes. Once trained, the model can make rapid predictions during runtime without needing to perform the actual entropy calculations, thereby resolving the contradiction between accuracy and complexity.
2Measurement precision
If entropy calculations are performed, then compression ratio prediction accuracy is improved, but processing time increases due to computational intensity
Solution Approach 1:
The patent substitutes the computationally intensive entropy calculation process with a pre-trained neural network inference process. The neural network, after being trained offline using entropy-based targets, can predict compression outcomes in constant time regardless of input size, eliminating the O(n log n) complexity of traditional entropy calculations while maintaining prediction accuracy.
3Productivity
If compression decision is made without predicting compression ratio, then processing speed is improved, but measurement precision of compression decision deteriorates
Solution Approach 1:
The patent introduces a neural network as an intermediary between the raw data and the compression decision. The network processes data through learned features and outputs a compressibility score that serves as a proxy for the actual compression ratio. This intermediary enables rapid decision-making while preserving accuracy by leveraging patterns learned during offline training with ground-truth compression ratios.
Data Source
AI summary
A data-compression analyzer can rapidly make a binary decision to compress or not compress an input data block or can use a slower neural network to predict the block's compression ratio with a regression model. A Concentration Value (CV) that is the sum of the squares of the frequencies and a Number of Zero (NZ) symbols are calculated from an un-sorted symbol frequency table. A rapid decision to compress is signaled when their product CV*NZ exceeds a horizontal threshold THH. During training, CV*NZ is plotted as a function of compression ratio C % for many training data blocks. Different test values of THH are applied to the plot to determine true and false positive rates that are plotted as a Receiver Operating Characteristic (ROC) curve. The point on the ROC curve having the largest Youden index is selected as the optimum THH for use in future binary decisions.