Backup Compression Skipping for High-Entropy Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing backup methods waste resources by attempting to compress high-entropy data, such as compressed file formats like .MP3 and .ZIP, which cannot be further compressed effectively, leading to inefficient use of memory and processing resources.
Innovation Solution
Identify high-entropy chunks during backup and automatically avoid compression of similar files by designating them as high-entropy files, using a compression threshold to determine when compression is not worth the resources required, thereby skipping subsequent compression attempts on similar files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If compression schemes are employed to compress all data during backup, then storage space is reduced and backup size is minimized, but memory and processing resources are wasted on high-entropy data that cannot be further compressed
Solution Approach 1:
The system performs preliminary analysis of data chunks during the backup process to identify high-entropy data before attempting compression. By detecting entropy levels in advance and skipping compression for identified high-entropy chunks, the system avoids wasting processing resources while still compressing compressible data, thus resolving the contradiction between minimizing backup size and conserving processing resources
Solution Approach 2:
The system applies different processing strategies to different parts of the backup data based on their entropy characteristics. Compressible low-entropy data undergoes compression to reduce backup size, while high-entropy data is identified and excluded from compression attempts. This localized quality approach ensures that compression resources are allocated only where effective, balancing storage optimization with resource conservation
2Volume of stationary object
If compression is attempted on all data, then storage efficiency is improved, but backup creation time increases due to processing overhead
Solution Approach 1:
The system performs preliminary entropy detection on data chunks before committing to compression operations. By identifying high-entropy data in advance and marking it for exclusion, the system avoids time-consuming compression attempts on data that would not benefit from compression, thus reducing overall backup creation time while still achieving storage efficiency on compressible data
Solution Approach 2:
The system applies compression selectively to only the portion of data that is likely to benefit from it (low-entropy compressible data), rather than attempting to compress all data. This partial action approach reduces the total processing time required for backup creation while still achieving significant storage space savings on the compressible portion of the data
Data Source
AI summary
Avoiding compression of high-entropy data during creation of a backup of a source storage. In one example embodiment, a method for avoiding compression of high-entropy data during creation of a backup of a source storage may include identifying a chunk with an original size in a file in a source storage, compressing, during creation of a backup of the source storage, the chunk to generate a compressed chunk with a compressed size, determining a compression ratio for the chunk by comparing the original size to the compressed size, determining whether the compression ratio is less than the compression threshold, and, in response to determining that the compression ratio is less than the compression threshold, automatically designating the file as a high-entropy file and automatically avoiding compression, during the creation of the backup, of chunks in a second similar file in the source storage.