Encrypted Data Storage with Tag-Based Duplicate Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently managing write transactions and ensuring data integrity and security in distributed storage networks, particularly in scenarios involving large volumes of data and complex tasks, while being resilient to failures and hacking attempts.
Innovation Solution
A distributed computing system that employs dispersed/distributed error encoding and decoding, along with secure data storage and task processing, utilizing a DSTN module for managing write transactions, error encoding, and integrity verification, ensuring data is stored and processed across multiple geographically diverse sites with robust security measures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in a buffer for processing, then task execution speed is improved, but data integrity and security are compromised due to potential corruption and hacking attempts
Solution Approach 1:
The data stored in the buffer is divided into multiple data slices that are distributed across different storage locations. This segmentation ensures that even if some slices are corrupted or compromised, the integrity of the overall data can be maintained through verification and reconstruction from remaining valid slices.
Solution Approach 2:
The system implements a verification mechanism that provides feedback on data integrity. The verification unit checks the integrity of data slices and can trigger reconstruction operations when corruption is detected, ensuring that only verified data is used for task execution.
2Reliability
If redundant copies of data are stored to ensure integrity, then data reliability is improved, but storage space and system complexity increase
Solution Approach 1:
Instead of storing complete redundant copies of data, the system segments data into slices and stores only necessary portions across multiple locations. This reduces the amount of redundant storage required while maintaining integrity through selective verification and reconstruction of only affected slices.
Solution Approach 2:
The system can discard corrupted or compromised data slices and recover the original data by reconstructing from remaining valid slices. This eliminates the need for full redundant copies, as only the corrupted portions need to be recovered rather than the entire data set.
3Object-affected harmful factors
If data is encrypted to prevent hacking, then security is improved, but processing efficiency and speed are reduced
Solution Approach 1:
The encryption is applied at the slice level rather than requiring encryption of the entire data set. This allows the system to process and verify individual slices independently, improving processing efficiency by enabling parallel operations on multiple slices while maintaining security through encrypted storage of each slice.
Solution Approach 2:
The system extracts only the necessary encrypted data slices for verification and processing, rather than handling the entire encrypted data set. This extraction approach reduces the computational overhead for security operations while maintaining the security benefits of encryption for the critical data portions.
Data Source
AI summary
A method includes storing a plurality of encrypted data, where each encrypted data of the plurality of encrypted data is generated based on encrypting corresponding underlying data. A plurality of data tags corresponding to the plurality of encrypted data are stored. First encrypted data and a first data tag are received, where the first encrypted data is generated based on encrypting first underlying data, and where the first data tag corresponds to the first underlying data of the first encrypted data. The first data tag is processed to determine whether the first underlying data is already stored as the corresponding underlying data of any of the plurality of encrypted data. The first encrypted data is stored based on determining the first underlying data is not already stored as other underlying data.