Container Image Executable De-duplication via Structural Delta Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Container images often contain duplicates or near-duplicates of data, which occupy unnecessary storage space and require inefficient de-duplication methods, especially when dealing with executable files where determining block sizes is challenging and performance is suboptimal.
Innovation Solution
The method involves splitting similar executable files into blocks based on their respective file structures, generating delta files by comparing blocks between updated and base executable files, and replacing corresponding files in the container image with delta files to reduce storage costs, while allowing transparent restoration of updated files during deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional de-duplication methods are used on container images, then storage space is reduced, but performance is suboptimal and determining block sizes is challenging
Solution Approach 1:
The patent segments executable files into variable-sized blocks based on their file structure (headers, sections, segments) rather than using fixed block sizes. This segmentation approach allows the system to identify and de-duplicate only the relevant portions of files, improving both storage efficiency and processing performance by avoiding unnecessary comparison of identical structural elements.
Solution Approach 2:
The patent changes the parameter of block size from fixed to variable, determined dynamically based on the file structure characteristics. By adjusting block boundaries to align with executable file structure (such as section headers and segments), the system achieves better de-duplication ratios while reducing the computational overhead of comparing only meaningful differences.
2Quantity of substance
If delta files are generated for updated executable files, then storage costs are reduced, but the complexity of managing block mappings increases
Solution Approach 1:
The patent creates delta files that contain only the differences between base and updated executable files. Instead of storing complete updated files, the system generates compact representations of changes (block mappings and differing data), significantly reducing storage requirements while maintaining the ability to reconstruct updated files when needed.
3Productivity
If duplicate data is stored in container images, then file restoration is simpler, but transmission time increases
Solution Approach 1:
The patent extracts and stores only the unique portions of updated executable files in delta format, separating the common base content (stored once in the base image) from the specific updates. This extraction approach reduces transmission time by sending only necessary differences while the restoration process combines these extracted differences with the base file to reconstruct the complete updated executable.
Data Source
AI summary
Methods, systems, and computer program products for de-duplicating data in executable files in a container image are disclosed. The method may include receiving a request to read a file in a first layer in a container image including a plurality of layers, wherein the file is a delta file which is from an updated executable file based on a base executable file, the base executable file is in a lower layer than the first layer in the container image, and the delta file includes block mappings between the updated executable file and the base executable file and different data between the two files, and blocks included in the two files are based on respective file structure. The method may also include restoring the updated executable file based on the delta file and the base executable file. The method may further include returning data in the updated executable file.