Entropy-Sorted Data Chunks for Higher Compression Density
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data compression into chunkfiles often results in inefficient space utilization, compression efficiency, and wasted computational resources due to random ordering of chunks, which affects tasks like malware detection and security in data platforms.
Innovation Solution
Utilizing entropy metrics to sort and reorganize data chunks based on their entropy values, allowing for improved compression algorithms to achieve higher compression rates and reduce storage space by positioning chunks with similar entropy values together.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data chunks are compressed into chunkfiles with random ordering, then compression is performed, but space utilization and compression efficiency are wasted
Solution Approach 1:
The patent applies preliminary action by sorting data chunks based on their entropy values before compression. The system calculates entropy for each chunk, orders them from lowest to highest entropy, and then compresses the sorted sequence. This preliminary sorting enables compression algorithms to achieve better compression ratios by processing chunks in an optimal order, directly resolving the contradiction between storage efficiency and compression efficiency.
2Loss of substance
If data chunks are sorted by entropy value, then compression efficiency improves, but computational resources and processing time increase
Solution Approach 1:
The patent applies parameter changes by using entropy value as a sorting parameter to optimize compression. Instead of arbitrary or random ordering, the system transforms the chunk ordering parameter to be based on entropy calculation results. This parameter transformation enables the system to achieve better compression ratios while the entropy calculation overhead is offset by the significant space savings in the compressed output.
3Reliability
If chunks are reorganized by entropy values, then pattern finding improves for malware detection, but device complexity increases
Solution Approach 1:
The patent applies local quality by creating localized patterns in the data stream through entropy-based sorting. Chunks with similar entropy values are grouped together, creating local homogeneity that enhances pattern recognition capabilities for malware detection. This localized organization of data with similar characteristics improves detection accuracy without requiring fundamental changes to the overall system architecture.
Data Source
AI summary
A method includes determining, by processing circuitry of a data platform, an entropy value for each of a plurality of data chunks stored by a storage system to obtain a corresponding plurality of entropy values. The method further includes reorganizing, by the processing circuitry and based on the corresponding plurality of entropy values, pointers referencing the plurality of data chunks into an ascending order or a descending order. The method also includes updating, by the processing circuitry, the pointers within the storage system based on the ascending order or the descending order. The method includes compressing, by the processing circuitry, the plurality of data chunks according to the order defined by the reorganized pointers to obtain a compressed chunkfile. The method additionally includes storing, by the processing circuitry, the compressed chunkfile superseding the plurality of data chunks.