Policy Driven Data Migration in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face challenges in securely and reliably storing data across multiple locations while being resistant to hacking and data loss, especially in the face of failures and hacking attempts.
Innovation Solution
A distributed computing system that employs dispersed error encoding and decoding, along with decentralized agreement protocols, to securely store and retrieve data across geographically diverse locations, ensuring data integrity and availability through redundant encoding and intelligent storage management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a distributed storage system, then data availability and redundancy are improved, but system complexity increases
Solution Approach 1:
The patent segments data into multiple shards that are distributed across different storage nodes. Each shard is independently stored and managed, allowing the system to maintain data availability even when individual nodes fail. This segmentation approach reduces the impact of single point failures while distributing the storage burden across multiple simpler nodes.
Solution Approach 2:
The patent introduces a coordinator as an intermediary component that manages the complex operations of data placement, retrieval, and failure detection. The coordinator abstracts the complexity from individual storage nodes, allowing them to operate simply while the coordinator handles the distributed coordination tasks, thus maintaining node simplicity while achieving high availability.
2Reliability
If error correction encoding is applied to data, then data integrity is improved, but storage space requirements increase
Solution Approach 1:
The patent divides data into smaller shards before applying error correction encoding. This segmentation allows error correction to be applied at a finer granularity, reducing the overhead per unit of data. By encoding smaller segments independently, the system achieves better integrity protection with lower space consumption compared to encoding large data blocks as a whole.
Solution Approach 2:
The patent dynamically adjusts error correction parameters based on the importance and size of data shards. Critical data receives stronger error correction, while less critical data uses lighter encoding. This parameter adaptation allows the system to optimize the balance between data integrity and storage space utilization, applying error correction only where most needed.
3Object-affected harmful factors
If decentralized agreement protocols are used, then security against hacking is improved, but operation time increases
Solution Approach 1:
The patent segments data operations into independent shard-level transactions that can proceed in parallel. Decentralized agreement protocols are applied at the shard level rather than at the entire dataset level, allowing multiple operations to occur simultaneously without sequential delays. This segmentation enables security verification to happen concurrently across multiple independent units, reducing total operation time while maintaining strong security.
Solution Approach 2:
The patent applies decentralized agreement protocols selectively to critical operations rather than all operations. For routine data access, the system uses faster centralized coordination where appropriate, while reserving full decentralized verification for write operations and data integrity-critical moments. This partial application approach maintains security where most needed while minimizing time loss from protocol overhead.
Data Source
AI summary
A method for storage units of a storage network begins by determining whether a first memory compares favorably to a predetermined performance level and determining whether a data access response activity associated with the first memory compares favorably to a data access response activity threshold. The method continues when the first memory compares unfavorably to the predetermined performance level and the data access response activity associated with the first memory compares unfavorably to a data access response activity threshold, by determining to migrate a set of encoded data slices stored in a first format from the first memory to a second memory, where a data object is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of encoded data slices. The method then continues, by selecting a second format for storing the set of encoded data slices and migrating the set of encoded data slices from the first memory to the second memory.