Layered Edge Storage With Erasure Coding for Atomic Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge computing systems face challenges in reducing the cost of operation of backend layer servers, maintaining consistency in data access, and minimizing latency, particularly in scenarios where edge servers lack sufficient storage capacity and resources.
Innovation Solution
A layered distributed storage (LDS) system is introduced, which employs a two-layer erasure-coded fault-tolerant distributed storage architecture. This system enables atomic access for read and write operations, optimizes communication and storage costs, and ensures fault-tolerance by using erasure codes such as minimum bandwidth regenerating (MBR) codes and random linear network codes (RLNC).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If edge servers store all files locally to provide fast access, then read latency is reduced, but storage capacity requirements increase significantly
Solution Approach 1:
The system divides the storage architecture into two segments: edge layer servers for frequently accessed files and backend layer servers for permanent storage. This segmentation allows read operations to be served from the fast edge layer when possible, while maintaining comprehensive storage capacity in the backend layer, thus resolving the contradiction between read latency and storage capacity requirements.
Solution Approach 2:
The patent introduces a vertical dimension to the storage hierarchy by adding the backend layer above the edge layer. This dimensional expansion creates a multi-layer storage architecture where data can be distributed across different levels based on access frequency, enabling fast local access for hot data while maintaining total storage capacity in the backend, thereby resolving the storage capacity vs. latency tradeoff.
2Loss of energy
If edge servers have limited storage capacity, then operational cost is reduced, but ability to store frequently accessed files locally deteriorates
Solution Approach 1:
The backend layer servers act as an intermediary storage layer between edge servers and permanent storage. When edge servers lack capacity for frequently accessed files, the system can retrieve these files from the backend layer, providing a middle ground that balances operational cost constraints with the need for fast data access, thus resolving the contradiction between operational cost and data access speed.
3Reliability
If strong consistency mechanisms are implemented across distributed servers, then data consistency is improved, but communication overhead and latency increase
Solution Approach 1:
The system implements local quality by providing strong consistency guarantees specifically for the edge layer where it is most needed for client operations, while using erasure coding for backend storage which provides adequate consistency with lower communication overhead. This differentiated approach to consistency based on location in the hierarchy resolves the contradiction between data consistency and communication latency.
Solution Approach 2:
The patent changes the consistency parameter from strong consistency to erasure coding for backend storage operations. Erasure coding provides sufficient data integrity and recovery capabilities with significantly reduced communication overhead compared to traditional strong consistency protocols, thus resolving the contradiction between data consistency reliability and communication latency.
4Quantity of substance
If backend layer servers are used for permanent storage, then storage capacity is increased, but communication cost between layers increases
Solution Approach 1:
The system performs preliminary actions by pre-coding data using erasure coding before transferring it to the backend layer. This preliminary encoding reduces the amount of data that needs to be communicated and stored redundantly, thereby decreasing communication cost between layers while maintaining adequate storage capacity in the backend, thus resolving the contradiction between storage capacity and communication cost.
Data Source
Figure 1
Figure 1A
Figure 2~2A
AI summary
A two-layer erasure-coded fault-tolerant distributed storage system offering atomic access for read and write operations is described. In some embodiments, a class of erasure codes known as regenerating codes (e.g. minimum bandwidth regenerating codes) for storage of data in a backend layer is used to reduce the cost of backend bulk storage and helps reduce communication cost of read operations, when a value needs to be recreated from persistent storage in the backend layer. By separating the functionality of edge layer servers and backend servers, a modular implementation for atomicity using storage-efficient erasure-codes is provided. Such a two-layer modular architecture permits protocols needed for consistency implementation to be substantially limited to interaction between clients and the edge layer, while protocols needed to implement erasure codes are substantially limited to the interaction between edge and backend layers.