Entropy Computation Using Precomputed Logarithmic Lookup Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face inefficiencies in determining the compressibility of data chunks, leading to suboptimal storage and resource utilization due to the lack of effective entropy computation methods for deciding on compression and deduplication.
Innovation Solution
A method involving the computation of entropy values using precomputed binary logarithmic values from a table, where counters denote the frequency of data items, allowing for efficient determination of compressibility and data reduction, with the option to store data chunks in either compressed or uncompressed form based on achieved data reduction thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional entropy computation methods are used to determine data compressibility, then compression decisions can be made, but the computation is inefficient and consumes excessive processing resources
Solution Approach 1:
The patent precomputes binary logarithmic values and stores them in a lookup table before entropy computation is needed. This preliminary action eliminates the need for complex logarithmic calculations during actual entropy computation, significantly reducing processing time and resource consumption while maintaining accuracy in compressibility determination
Solution Approach 2:
The patent replaces complex mathematical computations (logarithmic functions) with a simpler lookup-based system. By substituting the mechanical computation process with a table lookup operation, the system achieves the same entropy computation goal with minimal processing overhead, directly improving productivity while reducing energy loss
2Quantity of substance
If compression is applied to all data chunks, then storage space is reduced, but processing overhead increases and may not be worthwhile for incompressible data
Solution Approach 1:
The patent uses entropy computation as a feedback mechanism to determine whether compression is appropriate for each data chunk. By calculating entropy and comparing it against thresholds, the system receives feedback about data compressibility and makes intelligent decisions about whether to apply compression, optimizing storage space utilization while avoiding unnecessary processing overhead for incompressible data
Solution Approach 2:
The patent changes the parameter of compression application from a fixed rule (compress all data) to a dynamic decision based on entropy values. By using entropy as a parameter to control compression behavior, the system adapts to different data types and characteristics, reducing storage requirements for compressible data while minimizing processing overhead for incompressible data
Data Source
AI summary
Techniques for data processing may include: computing an entropy value for the chunk; determining, in accordance with the entropy value for the data chunk, whether the data chunk is compressible; and responsive to determining the data chunk is compressible based on the entropy value for the chunk, compressing the data chunk. The entropy value may be determined using counters for data items where the counters denote current frequencies of different allowable data items in the data chunk; and performing second processing using the counters to determine an entropy value for the data chunk, wherein said second processing includes selecting a precomputed binary logarithmic value from a table for each of the counters. The table may include integer representations of binary logarithmic values. The second processing may include loading multiple data items of the chunk into a register, extracting each data item from the register and incrementing a corresponding counter.