Kernel Checksum Verification for Storage Duplication Prevention

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing file systems face inefficiencies in preventing duplicate files from being downloaded and stored, leading to increased bandwidth and storage consumption, as well as unnecessary computational burdens from deduplication operations.

Innovation Solution

A modified kernel in the operating system uses checksums from a manifest file to determine if files already exist in the storage device, preventing duplicates by creating memory links or blocking downloads, thereby reducing redundant file transfers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If files are downloaded from remote computing device without duplication checking, then download speed and simplicity are improved, but bandwidth consumption and storage usage increase

Engineering Contradiction:

Improvedownload speedVSAvoidbandwidth consumption

Core Design Contradiction:

SpeedVSLoss of energy

Solution Approach 1:

The system performs preliminary actions by obtaining the manifest file with checksums before downloading files, and checking whether files already exist in storage by comparing checksums. This preliminary verification prevents unnecessary downloads, resolving the contradiction between download speed and bandwidth consumption by filtering out redundant transfers in advance.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The manifest file acts as an intermediary that contains checksum information for all files in the software package. This intermediary enables the system to verify file presence and integrity without directly comparing entire files, thus maintaining fast download speeds while reducing bandwidth consumption through efficient checksum-based verification.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If deduplication operations are performed on storage device, then storage space is optimized, but computational burden and processing time increase

Engineering Contradiction:

Improvestorage space utilizationVSAvoidcomputational burden

Core Design Contradiction:

Quantity of substanceVSPower

Solution Approach 1:

The system performs deduplication prevention in advance by checking the manifest file and comparing checksums before files are downloaded to storage. This preliminary action prevents duplicate files from being written to storage in the first place, achieving storage space optimization without the computational burden of post-download deduplication operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The download system serves itself by automatically checking the manifest file and identifying files that already exist in storage. This self-service mechanism prevents duplicates without requiring external deduplication operations, thus optimizing storage space while minimizing computational burden.

Inventive Principle:

Principle #25Self-service

3Reliability

If checksum verification is performed for each file, then file integrity and duplication prevention are improved, but processing time and complexity increase

Engineering Contradiction:

Improvefile integrity verificationVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Instead of verifying entire files, the system uses checksums (compact representations or copies of file content characteristics) to verify file integrity and detect duplicates. This copying approach maintains high reliability for detecting duplicates while significantly reducing processing time compared to full file comparisons.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system changes the verification parameter from comparing entire files to comparing checksum values. This parameter change maintains file integrity verification reliability while dramatically reducing processing time, as checksum comparison is computationally much less intensive than full file comparison.

Inventive Principle:

Principle #35Parameter changes

4Quantity of substance

If multiple software packages share common files, then storage efficiency is improved through deduplication, but network resources are wasted downloading duplicates

Engineering Contradiction:

Improvestorage efficiencyVSAvoidnetwork resource consumption

Core Design Contradiction:

Quantity of substanceVSLoss of energy

Solution Approach 1:

The system performs preliminary checking against the manifest file before initiating downloads, identifying files that are already present in storage. This preliminary action prevents network resources from being wasted on downloading duplicate files while ensuring storage efficiency is maintained across multiple software packages.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses feedback from the manifest file checksums to determine whether files already exist in storage. This feedback mechanism enables intelligent download decisions, preventing network resource waste on duplicates while maintaining storage efficiency across multiple software packages.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11829240B2Preventing duplication of files in a storage device

Publication Date: 2023.11.28 RED HAT LLC

AI summary

Duplication of files in a storage device of a computing device can be avoided using some techniques described herein. In one example, a system can determine a checksum of a file in a software package. The system can then determine that the file is absent from a storage device by issuing a command for accessing the file based on the checksum. In response to determining that the file is absent from the storage device, the system can download a copy of the file from a remote computing device to the storage device over a network.