Erasure Coding Large Data Objects Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in handling large data objects, particularly in providing responsive erasure coding solutions that minimize complexity and cost while ensuring data reliability.
Innovation Solution
The implementation of a distributed storage system that uses a combination of Type I and Type II data chunks, where Type II chunks are processed separately and efficiently stored with erasure coding tasks executed on nodes with local data copies to reduce network traffic, allowing for early acknowledgement of data storage requests and flexible handling of large data objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional erasure coding is used for large data objects, then data reliability is ensured, but processing time and network traffic increase significantly
Solution Approach 1:
The patent segments large data objects into multiple smaller chunks, allowing parallel processing of erasure coding operations on each chunk independently. This reduces the overall processing time while maintaining data reliability through distributed storage of coded chunks across the network
Solution Approach 2:
The system performs preliminary actions by pre-processing and preparing data chunks before the actual erasure coding operation. This includes validating chunk integrity and pre-establishing storage locations, which streamlines the subsequent coding process and reduces total processing time
2Reliability
If traditional erasure coding is used for large data objects, then data reliability is ensured, but network traffic increases
Solution Approach 1:
By dividing large data objects into smaller chunks, the system reduces the amount of data that needs to be transmitted over the network at any given time. This segmentation enables more efficient network utilization and reduces overall network traffic while maintaining the same level of data reliability through distributed chunk storage
Solution Approach 2:
The patent introduces intermediary storage nodes that temporarily hold data chunks during the erasure coding process. These intermediaries reduce direct network traffic between source and destination by providing local caching and buffering capabilities
3Device complexity
If large data objects are processed as a single unit, then simplicity is maintained, but processing efficiency decreases
Solution Approach 1:
The system automatically segments large data objects into manageable chunks without requiring complex manual configuration. This segmentation enables parallel processing across multiple storage nodes, significantly improving processing efficiency while the automated chunk management keeps operational complexity low
Solution Approach 2:
The patent merges multiple independent chunk processing operations into a unified erasure coding workflow. This allows the system to maintain simple, modular processing units while achieving high overall efficiency through coordinated parallel execution of multiple chunks
Data Source
AI summary
A system, computer program product, and computer-executable method for use with a distributed storage system comprising a plurality of storage nodes each having attached storage devices, the system, computer program product, and computer-executable method including receiving a request, at a first storage node of the plurality of storage nodes, to store a large portion of data, using at least one of a first type of data chunk and a plurality of a second type of data chunks to store the large portion of data, processing each of the plurality of the second type of data chunks, processing each of the at least one of the first type of data chunk, and returning an acknowledgement to the request.