Compressed Data Verification Levels for Storage Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data compression can lead to data corruption during storage, resulting in valid data loss if the compressed data is corrupted before being verified, as the original and uncompressed versions are discarded once the compressed counterpart is stored to disk.
Innovation Solution
A verification process is implemented where compressed data is read from a global cache, decompressed based on selected verification levels, and compared to its original version to ensure integrity, with different verification levels requiring distinct resources and response times, and only verified data is written to disk.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is performed to reduce storage capacity, then storage efficiency is improved, but data integrity is worsened due to potential corruption during storage
Solution Approach 1:
The patent performs verification of compressed data blocks before they are permanently written to storage devices. The system reads compressed data from cache, verifies its integrity using checksums or cyclic redundancy checks (CRC), and only writes verified blocks to disk. This preliminary verification action prevents corruption from propagating to permanent storage.
Solution Approach 2:
The patent introduces a verification layer as an intermediary between compression and permanent storage. This intermediary process includes reading compressed blocks from cache, computing verification values, comparing them against expected values, and conditionally writing to disk based on verification results. The intermediary verification step isolates the compression process from direct storage exposure.
2Reliability
If verification processes are performed on compressed data, then data integrity is improved, but processing time is worsened
Solution Approach 1:
The patent divides verification into different levels: fast verification using checksums for quick validation, and slower verification using CRC or more thorough checks when needed. The system segments the verification process into multiple passes with different thoroughness levels, allowing quick rejection of obviously corrupted blocks while performing more intensive verification only when necessary.
Solution Approach 2:
The patent dynamically adjusts verification parameters such as the type of check performed (checksum vs. CRC), the number of blocks verified, and the threshold for re-verification based on observed error rates and system conditions. When error rates are low, lighter verification is used; when errors are detected, more intensive verification is triggered, optimizing the balance between integrity and performance.
Data Source
AI summary
Embodiments of the present disclosure relate to verifying compressed data. Compressed data files can be read from a global cache for a storage device into a local buffer. A data verification level of a plurality of data verification levels can be selected to perform on the compressed data files. An amount of data blocks of each data file can be decompressed based on the determined data verification level. An integrity of the compressed data files verified using the decompressed data blocks.