Dispersed Storage Network Data Loss Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed storage systems face challenges in maintaining data integrity and availability due to storage unit failures, requiring redundant copies and complex error correction schemes, which can lead to data loss and security vulnerabilities.

Innovation Solution

A dispersed storage network (DSN) utilizing a managing unit, integrity processing unit, and computing devices with a dispersed storage error encoding and decoding mechanism, employing Cauchy Reed-Solomon encoding to split data into encoded slices stored across multiple geographically diverse sites, ensuring data recovery without redundant copies and enhanced security.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If redundant copies of data are stored in distributed storage systems, then data availability and reliability are improved, but storage space consumption increases and security vulnerabilities arise

Engineering Contradiction:

Improvedata availabilityVSAvoidstorage space consumption

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments data into multiple encoded slices using error correction codes, where each slice is stored separately across different storage units. This allows data recovery from any sufficient number of slices without requiring redundant copies of the entire data set, thus improving reliability while reducing storage space consumption.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the storage parameter from storing complete data copies to storing encoded slices. By using error correction codes, the system can reconstruct data from a subset of slices, transforming the reliability mechanism from redundancy-based to error-correction-based, which reduces storage requirements.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If complex error correction schemes are implemented, then data integrity is improved, but system complexity increases

Engineering Contradiction:

Improvedata integrityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent uses error correction codes to create encoded copies of data slices. These encoded slices can be independently stored and transmitted, simplifying the error correction process compared to complex schemes. The decoding process automatically reconstructs the original data from sufficient slices, maintaining integrity without requiring complex management mechanisms.

Inventive Principle:

Principle #26Copying

3Reliability

If data is stored across multiple geographically diverse sites, then data availability and security are improved, but detection and measurement of data loss events becomes more difficult

Engineering Contradiction:

Improvedata availabilityVSAvoiddata loss event detection

Core Design Contradiction:

ReliabilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent implements a feedback mechanism where storage units periodically report their status to the managing unit. The managing unit maintains a mapping between data slices and storage units, enabling it to detect when slices are missing or corrupted. This feedback loop allows for timely detection of data loss events across geographically distributed sites without increasing operational complexity.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10114698B2Detecting and responding to data loss events in a dispersed storage network

Publication Date: 2018.10.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method for use in a dispersed storage network operates to identify missing, out-of-date or otherwise compromised encoded data slices in a dispersed storage network (DSN), and when a decode threshold of encoded data slices is not available to rebuild an associated data object, determine whether a data loss event has occurred. When a data loss event is determined to have occurred the method continues by initiating a process to recover all or some of the lost data and may include notification to DSN entities that a data loss event has occurred.