Dispersed Storage Metadata Encoding for Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges with data integrity and security due to the failure of physical movement-based memory devices, such as disc drives, which can lead to data loss and unauthorized access, especially as the amount of data grows, and the overhead of redundant arrays like RAID becomes inefficient.
Innovation Solution
A distributed storage network system that uses error coding dispersal storage to partition data into slices, which are then encoded and stored across multiple physically diverse locations, allowing for secure and reliable data retrieval even in the event of device failures, with a registry system for managing and authenticating access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant array of independent discs (RAID) is used to protect against disc drive failure, then data reliability is improved, but device complexity and storage overhead increase significantly
Solution Approach 1:
The patent segments data into multiple data slices and disperses them across different storage locations using error coding. Instead of replicating entire data blocks across multiple discs as in RAID, the system divides data into segments and stores them distributedly, reducing the complexity of managing redundant arrays while maintaining reliability through error correction codes.
Solution Approach 2:
The patent replaces the mechanical redundancy system of RAID (which requires multiple physical disc drives and complex array management) with an error coding-based dispersal system. This substitution eliminates the need for complex RAID controller hardware and software management, reducing device complexity while achieving similar or better reliability through mathematical error correction.
2Reliability
If multiple copies of data are stored to prevent data loss, then data reliability is improved, but storage capacity is reduced due to redundancy overhead
Solution Approach 1:
The patent changes the parameter of data representation by applying error coding transformations to data slices. Instead of storing raw redundant copies, the system encodes data with error correction information, allowing recovery from failures while using storage capacity more efficiently. The error coding enables the system to tolerate failures without requiring full redundant copies of the original data.
3Ease of manufacture
If physical movement-based memory devices are used for storage, then ease of manufacture is improved, but data integrity deteriorates due to bit level corruption and device failure
Solution Approach 1:
The patent applies error coding to data slices before storage, creating a protective cushion against potential bit corruption and device failures. This preliminary error correction encoding ensures that even if physical degradation occurs during storage, the original data can be recovered, cushioning against the inherent unreliability of physical movement-based memory devices.
4Quantity of substance
If disc drives are used for secondary memory storage, then cost is reduced compared to higher-grade drives, but reliability deteriorates due to routine bit level corruption and complete failure
Solution Approach 1:
The patent creates encoded copies of data slices with error correction information embedded. Instead of using expensive high-reliability drives, the system stores data on cheaper disc drives but protects the data by creating encoded copies that can withstand corruption and failure. This allows the use of low-cost storage media while maintaining reliability through the copying and error correction mechanism.
Data Source
AI summary
A method begins by a processing module dispersed storage error encoding a data segment to produce a set of encoded data slices and dispersed storage error encoding metadata associated with the data segment to produce a set of encoded metadata slices. The method continues with the processing module creating a set of data slice names for the set of encoded data slices and creating a set of metadata slice names based on the set of data slice names. The method continues with the processing module sending the set of encoded data slices and the set of data slice names to a dispersed storage network (DSN) memory for storage therein. The method continues with the processing module sending the set of encoded metadata slices and the set of metadata slice names to the DSN memory for storage therein.