Clustered Data Storage Layout to Cut Erasure Coding Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current decentralized storage solutions generate excessive data traffic due to performing encryption and erasure coding at the client side, leading to congestion on network links and increased demand for additional network equipment.
Innovation Solution
Spatially separate encryption and erasure coding, with encryption performed at the client and erasure coding conducted within a cluster of network nodes, such as an operator's network, and send original data between clusters for storage, reducing the volume of data transferred.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If encryption and erasure coding are performed at the client side, then data security and reliability are improved, but data traffic volume increases excessively causing network congestion
Solution Approach 1:
The patent segments the data processing function by separating encryption (performed at client side) from erasure coding (performed at storage node side). This segmentation allows the client to only upload encrypted data without performing erasure coding, thereby reducing the data traffic volume while maintaining both data security and reliability through the combination of encryption and distributed erasure coding across multiple storage nodes
Solution Approach 2:
The storage node acts as an intermediary that performs erasure coding on the encrypted data received from the client. This intermediary function allows the system to maintain data security (since encryption is done at client) while reducing network traffic (since erasure coding is done at storage node rather than expanding data before transmission)
2Reliability
If erasure coded chunks are sent between clusters for storage, then storage distribution and reliability are improved, but computation and communication overhead increases
Solution Approach 1:
The patent applies local quality by allowing each cluster to independently perform erasure coding on encrypted data and store locally-generated erasure coded chunks. This local processing reduces inter-cluster communication overhead while maintaining distributed storage reliability, as each cluster operates autonomously on its portion of the data without requiring extensive coordination with other clusters
Data Source
AI summary
Methods and apparatus are provided. In an example aspect, a method of managing data storage in a communication system is provided. The communication system includes a plurality of clusters of network nodes, and the network nodes in each cluster of network node include one or more data storage nodes. The method comprises receiving data to be stored from a data source node, generating, from the data to be stored, a first subset of erasure coded chunks, causing the first subset of erasure coded chunks to be stored in one or more of the data storage nodes in at least one first cluster of the plurality of clusters, and sending, to a second cluster of the plurality of clusters, the data to be stored.