Matrix-Based Erasure Coding for Fast Distributed Data Reconstruction

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing storage systems with error correction code redundancy schemes have single points of failure and cannot reconstruct failed drives quickly, as they do not employ distributed hash tables and disconnect responsibility for storage from maintenance.

Innovation Solution

A dynamically scalable redundant distributed storage system that separates logical responsibility for data from actual control, using distributed hash tables and components like file and storage managers, and implements error correction code methods or replication for robust failure recovery.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a single component is used for both storage responsibility and maintenance, then the system structure is simple, but the system has single points of failure and cannot reconstruct failed drives quickly

Engineering Contradiction:
Improvefault toleranceVSAvoidsystem structure
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent divides the storage system into separate components: file managers that handle logical responsibility for data and storage managers that handle actual data maintenance. This segmentation eliminates single points of failure by distributing responsibilities across multiple specialized components, allowing the system to tolerate component failures while maintaining data availability through the separation of concerns between file managers and storage managers.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces file managers as intermediary components between clients and storage managers. File managers handle logical operations and coordinate with storage managers for actual data maintenance, creating a mediator layer that improves reliability by preventing single points of failure while managing the complexity through standardized interfaces and protocols.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Reliability

If error correction codes are implemented in traditional storage systems, then data integrity is improved, but the reconstruction time for failed drives increases due to centralized control

Engineering Contradiction:
Improvedata integrityVSAvoidreconstruction time
Core Design Contradiction:
ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the error correction and data reconstruction process across multiple storage managers rather than concentrating it in a single component. Each storage manager independently handles maintenance of its assigned data blocks, allowing parallel reconstruction operations that reduce overall reconstruction time while maintaining data integrity through distributed error correction capabilities.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent implements preliminary error correction coding during the initial data storage phase, distributing encoded data across multiple storage managers. This preliminary action enables faster reconstruction during failures because the error correction capability is already embedded in the distributed data structure, eliminating the need for time-consuming centralized computation during recovery.

Inventive Principle:
Principle #10Preliminary action

3Productivity

If distributed hash tables are not used, then the system structure is simpler, but the system cannot efficiently locate and reconstruct data across distributed storage elements

Engineering Contradiction:
Improvedata location efficiencyVSAvoidsystem architecture
Core Design Contradiction:
ProductivityVSDevice complexity

Solution Approach 1:

The patent implements distributed hash tables that serve multiple functions: locating data blocks across storage managers, coordinating error correction operations, and managing the mapping between logical data and physical storage. This multi-functional approach improves productivity by enabling efficient data location while managing architectural complexity through a unified distributed coordination mechanism.

Inventive Principle:
Principle #6Universality (Multi-functionality)

Solution Approach 2:

The distributed hash table acts as an intermediary layer between file managers and storage managers, providing efficient data location capabilities without requiring complex peer-to-peer communication protocols. This mediator structure improves productivity by enabling fast data retrieval while managing the complexity of distributed coordination through a standardized hash-based lookup mechanism.

Inventive Principle:
Principle #24Intermediary (Mediator)

Data Source

PatentUS10536167B2Matrix-based error correction and erasure code methods and system and applications thereof
Publication Date: 2020.01.14 FILE SYSTEM LABS LLC
  • US10536167B2 patent drawing
  • US10536167B2 patent drawing
  • US10536167B2 patent drawing

AI summary

A distributed data storage system breaks data into n slices and k checksums using at least one matrix-based erasure code based on matrices with invertible submatrices, stores the slices and checksums on a plurality of storage elements, retrieves the slices from the storage elements, and, when slices have been lost or corrupted, retrieves the checksums from the storage elements and restores the data using the at least one matrix-based erasure code and the checksums. In a method for ensuring restoration and integrity of data in computer-related applications, data is broken into n pieces, k checksums are calculated using at least one matrix-based erasure code based on matrices with invertible submatrices, and the n data pieces and k checksums are stored on n+k storage elements or transmitted over a network. If, upon retrieving the n pieces from the storage elements or network, pieces have been lost or corrupted, the checksums are retrieved and the data is restored using the matrix-based erasure code and the checksums.