Distributed Data Storage via Segmentation and Shuffling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems are inadequate in protecting data from unauthorized access and loss, particularly for sensitive information like medical data, where privacy and availability must be ensured.
Innovation Solution
Data is divided into portions, shuffled, and distributed across geographically diverse storage resources, with a reassembly key provided to authorized users, allowing reconstruction from fewer than all portions due to redundancy, thus preventing unauthorized access and loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored on a single device, then access is simple and fast, but data loss risk increases if the device is damaged or stolen
Solution Approach 1:
The data is divided into multiple portions and distributed across different storage devices. This segmentation ensures that if one device fails or is stolen, the data can still be recovered from the remaining devices, thereby improving reliability without requiring a single complex centralized system.
Solution Approach 2:
Multiple copies of the data portions are created and stored on different devices. This copying mechanism provides redundancy, allowing the system to maintain data availability even when individual storage devices fail, thus improving reliability through duplication rather than complex replication systems.
2Reliability
If data is divided into portions and distributed across multiple devices, then data loss protection improves, but data retrieval complexity increases
Solution Approach 1:
A centralized indexing system acts as an intermediary between the user and the distributed storage devices. This indexer maintains a mapping of data portions to their locations across different devices, allowing users to retrieve data simply by querying the indexer rather than manually coordinating multiple storage devices, thus improving ease of operation.
3Reliability
If redundancy is added to data portions, then data recovery capability improves, but storage space requirements increase
Solution Approach 1:
The data is segmented into portions with strategic redundancy, where not all portions need to be stored on every device. This selective segmentation approach provides sufficient recovery capability while minimizing redundant storage space compared to storing complete copies on all devices.
Solution Approach 2:
Different storage devices hold different portions of the data with varying redundancy levels tailored to their specific roles and failure probabilities. This local quality approach optimizes storage space by applying redundancy only where and to the extent necessary, rather than uniformly across all data and devices.
Data Source
AI summary
In one example, data may be divided into blocks, and the blocks may be stored in various storage resources. Data to be stored may be provided to a data divider. The data divider may divide the data into blocks. Redundancy may be introduced into the blocks to allow the original data to be reconstructed at some point in the future, even if fewer than all of the original blocks are available. The blocks may be sent to a data shuffler, which may shuffle the blocks, and may provide a key that describes how to reconstruct the original data from the blocks. The key may be provided to the owner of the data. When the original data is to be retrieved, the key may be provided to a reassembler, which retrieves the blocks from the various storage resources and reconstructs the data using the key.