Inline Erasure Coding With Key Mapping for Distributed Log Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise storage systems face challenges in scalability, flexibility, and management overhead due to their centralized nature, leading to high upfront costs and inefficiencies in adapting to changing application requirements, and they lack effective fault tolerance mechanisms.
Innovation Solution
A distributed storage system implementing erasure coding with a key-to-physical medium map that acts as an intermediary between the file system and storage medium, allowing for efficient data storage and retrieval while providing fault tolerance by replicating the key-to-physical medium map across multiple disks, thus avoiding single points of failure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If centralized storage appliances are used, then storage capacity and performance can be increased, but device complexity and management overhead increase significantly
Solution Approach 1:
The patent divides the centralized storage system into multiple independent distributed storage nodes that can operate autonomously. Each node maintains its own storage capacity while the collective network provides scalable storage without centralized management overhead. This segmentation allows capacity to grow by simply adding more independent nodes rather than managing increasingly complex centralized appliances.
2Ease of manufacture
If proprietary storage appliances are integrated, then storage functions are unified, but adaptability to different application requirements decreases
Solution Approach 1:
The distributed storage system employs dynamic configuration where storage policies, data placement, and node participation can be adjusted in real-time based on application requirements. Unlike static proprietary appliances, the system can dynamically adapt to different workloads, performance needs, and storage requirements by reconfiguring the distributed network without hardware changes.
Solution Approach 2:
The patent creates a universal storage platform where the same distributed storage nodes can serve multiple different applications and workloads simultaneously. The system provides multi-functionality by allowing a single distributed infrastructure to handle various storage requirements (hot, warm, cold storage; archival; caching; etc.) through software-defined policies rather than dedicated appliances for each function.
3Reliability
If erasure coding is implemented across distributed storage devices, then fault tolerance is improved, but computational complexity increases
Solution Approach 1:
The erasure coding implementation in the distributed system is designed to be self-managing, where each node independently performs encoding and decoding operations on its local data without requiring centralized computational coordination. This self-service approach distributes the computational complexity across all nodes rather than concentrating it, making the fault tolerance mechanism scalable and reducing the burden on any single component.
Data Source
AI summary
Presented herein are methodologies for implementing erasure coding in a distributed log structured storage system. A method includes receiving a write request for first data from a file system, selecting a physical sector on a selected storage device in an array of storage devices on which to store the first data, assigning a key to the physical sector, storing the key and an indication of the physical sector in a key-to-physical medium map, erasure coding the data, including generating parity data associated with the first data, writing the first data and the parity data as a data stripe to each storage device in the array of storage devices and, in response to receiving the write request, sending the key to the file system. Read, update, and delete procedures in the context of a log structured framework are also described.