Distributed Storage Deduplication and Erasure Coding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

NoSQL/SQL database clusters face performance degradation and inefficient storage due to heavy loading of single storage nodes when deduplicated data is stored, leading to bottlenecks and inability to scale out performance.

Innovation Solution

A distributed storage system where compute nodes instruct storage nodes to perform deduplication and erasure coding across multiple storage nodes, reducing the load on individual storage nodes and improving storage efficiency by spreading data redundancy across multiple nodes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is multiplexed (triplicated) for protection purposes, then data reliability is improved, but storage efficiency deteriorates due to increased storage requirements

Engineering Contradiction:

Improvedata protectionVSAvoidstorage capacity

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent changes the redundancy parameter from full triplication (3x storage) to erasure coding with configurable redundancy ratios. By adjusting the coding parameters (k,m values), the system can achieve different levels of protection with optimized storage overhead, resolving the contradiction between reliability and storage efficiency

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent combines multiple data protection mechanisms (deduplication at compute nodes + erasure coding at storage nodes) to create a composite protection system. This layered approach achieves reliable data protection while minimizing total storage requirements compared to simple triplication

Inventive Principle:

Principle #40Composite materials

2Quantity of substance

If deduplicated data is stored in a single storage node, then storage efficiency is improved, but system performance deteriorates due to concentrated access load creating bottlenecks

Engineering Contradiction:

Improvestorage efficiencyVSAvoidsystem performance

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent segments the storage of deduplicated data across multiple storage nodes using erasure coding. Instead of concentrating all deduplicated data on one node, the system divides data into chunks and distributes them across multiple nodes with redundancy, eliminating the single-node bottleneck while maintaining storage efficiency

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent transitions from a single-dimension storage model (one storage node) to a multi-dimensional distributed storage model. By adding the dimension of distribution across multiple nodes while maintaining deduplication benefits, the system achieves both storage efficiency and performance scalability

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11768608B2Computer system, compute node, and data management method

Publication Date: 2023.09.26 HITACHI VANTARA LTD

AI summary

A computer system includes a NoSQL/SQL cluster and a distributed storage. In order to make storage target data redundant, a compute node in the NoSQL/SQL cluster instructs other compute nodes in the NoSQL/SQL cluster to write the storage target data into the distributed storage. As regards a file containing the storage target data, the compute node in the NoSQL/SQL cluster deduplicates data in storage apparatuses of a plurality of storage nodes in the distributed storage. The distributed storage performs erasure coding to store a file of the storage target data newly found to be duplicate in deduplication and to store a file of the storage target data not deduplicated in deduplication.