Data Ingestion Sorting With Error-Encoded Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud data storage systems face challenges in efficiently managing and storing large volumes of data across distributed networks, particularly in ensuring data integrity and availability without redundant copies, while maintaining security and scalability.
Innovation Solution
A dispersed storage network (DSN) architecture that uses error encoding techniques, such as Cauchy Reed-Solomon encoding, to distribute data across multiple storage units, allowing for efficient error correction and data retrieval even in the presence of failures, with a managing unit and integrity processing unit to manage and rebuild encoded data slices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple storage units using error encoding, then data availability and reliability are improved, but system complexity increases
Solution Approach 1:
The patent segments data into multiple data slices that are distributed across different storage units. Each data slice is stored independently, allowing the system to retrieve data even if some storage units fail. This segmentation approach directly improves data availability while managing complexity through modular organization.
Solution Approach 2:
The patent introduces an intermediary encoding layer that transforms original data into encoded data slices before distribution. This intermediary step using error correction codes enables reliable data retrieval from distributed storage units without requiring redundant copies, resolving the contradiction between reliability and complexity.
2Reliability
If redundant copies of data are stored, then data availability is improved, but storage efficiency deteriorates
Solution Approach 1:
The patent extracts the redundancy requirement from traditional backup approaches and replaces it with error correction encoding. Instead of storing complete redundant copies of data, the system extracts essential redundancy information through encoding, distributing data slices across storage units without duplicating entire datasets. This achieves data availability while maintaining storage efficiency.
Solution Approach 2:
The patent changes the parameter of data representation from original form to encoded form. By transforming data through error correction encoding, the system achieves the same reliability goal as redundant copying but with reduced storage requirements. The encoded data slices contain distributed information that can be reconstructed without storing full redundant copies.
3Reliability
If data is encrypted and distributed, then data security is improved, but processing speed deteriorates
Solution Approach 1:
The patent performs preliminary encryption and encoding of data before distribution to storage units. By pre-processing data with encryption and error correction encoding before storage, the system ensures data security during transmission and storage without requiring real-time processing during retrieval. This preliminary action resolves the contradiction by shifting processing overhead to the storage phase rather than the retrieval phase.
Data Source
AI summary
A method begins with a first group of computing devices of a plurality of computing devices of a storage network receiving data objects having a first data type identifier of a plurality of data type identifiers, where the plurality of data type identifiers correspond to a plurality of data types associated with the data objects. The method continues with the first group of computing devices interpreting the data objects having the first data type identifier to sort, based on sorting criteria the data objects into a first processing category and a second processing category. The method continues with the first group of computing devices error encoding the data objects in the second processing category based on short term storage error encoding parameters to produce pluralities of sets of encoded data slices and sending the pluralities of sets of encoded data slices to storage and execution units for storage therein.