Two-Layer Erasure-Coded Storage for Edge Consistency Costs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge computing systems face challenges in maintaining strong consistency and reducing operational costs due to limited resources and geographic distance between edge and backend servers, leading to inefficiencies in data access and storage.
Innovation Solution
A layered distributed storage system using erasure codes, such as minimum bandwidth regenerating (MBR) codes, to ensure atomic access and fault-tolerance, allowing clients to interact only with edge servers and optimizing communication and storage costs by separating functionalities between edge and backend layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If strong consistency mechanisms are implemented in edge computing systems, then data consistency is improved, but operational cost increases due to frequent backend layer communication
Solution Approach 1:
The system segments storage functionality into two layers: edge layer servers for frequently accessed data and backend layer servers for permanent storage. This segmentation allows consistency operations to be localized at the edge layer, reducing the frequency of backend communication and thereby lowering operational costs while maintaining data consistency through erasure-coded replication across edge servers.
Solution Approach 2:
The edge layer servers act as intermediaries between clients and the backend layer. They implement erasure-coded storage locally, handling read/write operations and consistency maintenance without requiring frequent backend intervention. This intermediary role reduces backend communication overhead and operational costs while preserving strong consistency guarantees.
2Speed
If all data is stored at the edge layer, then access speed is improved, but storage capacity is insufficient for massive data volumes
Solution Approach 1:
The storage system is segmented into edge layer storage for hot data requiring fast access and backend layer storage for cold data and permanent archival. Erasure coding distributes data across edge servers, enabling fast local access to frequently used portions while maintaining the ability to retrieve any data from the backend when needed, thus balancing speed and capacity.
Solution Approach 2:
The system adds a temporal dimension to data placement by using erasure codes to reconstruct data dynamically. Frequently accessed data resides at the edge for immediate retrieval, while less frequently accessed data can be reconstructed from coded fragments stored at the backend, effectively extending storage capacity without compromising access speed for hot data.
3Volume of stationary object
If backend layer servers are used for permanent storage, then storage capacity is improved, but communication cost and operational expense increase
Solution Approach 1:
The system performs preliminary erasure coding at the edge layer before data is transferred to the backend. This preliminary action creates redundant coded fragments that can be used for future data reconstruction, reducing the need for frequent backend communication and thereby lowering communication costs while maintaining adequate storage capacity.
Solution Approach 2:
The system changes the parameter of data representation by storing erasure-coded fragments instead of complete data copies at the backend. This parameter change reduces the amount of data that needs to be communicated with the backend layer, lowering communication costs while preserving the ability to reconstruct original data when needed.
4Quantity of substance
If edge servers have finite storage capacity, then resource efficiency is improved, but ability to store all files permanently is reduced
Solution Approach 1:
The system creates coded copies of data fragments across multiple edge servers using erasure coding. Instead of storing complete copies of all files at every edge server, the system distributes encoded fragments, allowing any sufficient subset of edge servers to reconstruct any file. This copying strategy maximizes storage resource efficiency while maintaining data availability and reliability.
Data Source
AI summary
A two-layer erasure-coded fault-tolerant distributed storage system offering atomic access for read and write operations is described. In some embodiments, a class of erasure codes known as regenerating codes (e.g. minimum bandwidth regenerating codes) for storage of data in a backend layer is used to reduce a cost of backend bulk storage and helps in reducing communication cost of read operations, when a value needs to be recreated from persistent storage in the backend layer. By separating the functionality of edge layer servers and backend servers, a modular implementation for atomicity using storage-efficient erasure-codes is provided. Such a two-layer modular architecture permits protocols needed for consistency implementation to be substantially limited to the interaction between clients and an edge layer, while protocols needed to implement erasure code are substantially limited to interaction between edge and backend layers.