Verifiable Codec Integrity Checks for Dispersed Storage Slices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current dispersed storage systems face inefficiencies in processing and verifying the integrity of encoded data, particularly due to the high resource consumption required for reverse functions like decompression and decryption before integrity checks can be performed.
Innovation Solution
The implementation of a dispersed storage network (DSN) with a managing unit, integrity processing unit, and computing devices that utilize error encoding and decoding functions like Cauchy Reed-Solomon encoding, allowing for secure and efficient storage and retrieval of data across multiple geographically distributed storage units, with built-in mechanisms for rebuilding 'bad' or missing encoded data slices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If integrity checks are performed after decompression and decryption, then data integrity can be verified, but processing resources are consumed excessively
Solution Approach 1:
The patent applies preliminary action by computing and storing integrity values (checksums) during the encoding phase, before compression and encryption operations. This allows integrity verification to be performed directly on encoded data slices without requiring reverse functions, thereby reducing processing resource consumption while maintaining reliability
Solution Approach 2:
The patent extracts the integrity verification function from the post-decoding process and implements it independently during the encoding phase. By separating integrity computation from the main data processing workflow, the system can verify integrity without consuming excessive processing resources during retrieval operations
2Reliability
If reverse functions (decompression, decryption) are performed before integrity checks, then accurate integrity verification is possible, but processing time increases
Solution Approach 1:
The patent performs integrity value computation as a preliminary action during the encoding phase, storing these values with the encoded data slices. This eliminates the need to perform time-consuming reverse functions before integrity checks, as verification can be done directly on the encoded data using the pre-computed integrity values
Solution Approach 2:
The patent segments the integrity verification process from the main data processing workflow by implementing independent integrity computation and verification mechanisms. This allows integrity checks to be performed on individual encoded data slices without requiring full decompression and decryption, thereby reducing processing time
3Reliability
If data is compressed and encrypted before encoding, then security and space efficiency are improved, but integrity verification becomes more complex
Solution Approach 1:
The patent applies preliminary action by computing integrity values during the encoding phase, before compression and encryption are applied to the final data slices. This ensures that integrity verification remains simple and direct, while still maintaining security through subsequent compression and encryption layers
Solution Approach 2:
The patent implements a universal integrity verification mechanism that works independently of the compression and encryption algorithms used. The integrity value computation and verification process is algorithm-agnostic, allowing the system to maintain data security through multiple transformation layers while keeping integrity verification complexity low
Data Source
AI summary
A method includes retrieving a decode threshold number of encoded data slices, wherein codecs process, in an order, a data segment and the processed data segment is encoded into a set of encoded data slices. The method further includes decoding the decode threshold number of encoded data slices to recover the processed data segment. In a reversed order to the order, applying a first codec on the processed data segment to produce a first partially processed recovered data segment. When the first codec is a verifiable codec, the method further includes separating the first partially processed recovered data segment into an initial integrity value and a processed data segment. The method further includes calculating a new integrity value from the processed data segment. When the new integrity value substantially matches the initial integrity value, the method further includes indicating that the set of encoded data slices is authentic.