Secure Data Deletion in HDFS via Random Disk Rotation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large distributed data platforms like Hadoop Distributed File System (HDFS) face challenges in securely deleting and sanitizing sensitive information, as existing solutions are inadequate, requiring destructive deletion or forced downtime, and data can be recovered using common recovery tools due to incomplete sanitization.
Innovation Solution
The solution involves modifying Apache Hadoop common HDFS source code to provide direct disk access for secure file deletion using a configurable number of passes with AES-256 bit hashing and random disk rotation, ensuring that deleted data is overwritten multiple times, making recovery impossible, and constantly shuffling data to sanitize both used and free disk space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If destructive deletion or forced downtime is used to sanitize data, then data security is improved, but system availability and operational continuity deteriorate
Solution Approach 1:
The patent segments the data sanitization process into individual file-level operations rather than platform-wide destructive deletion. Each file is sanitized independently by identifying its distributed blocks and overwriting them with random data, allowing the system to maintain operational continuity while achieving secure deletion of specific sensitive files without taking the entire platform offline.
Solution Approach 2:
The system performs preliminary identification of all blocks containing the file to be deleted before actual sanitization begins. By pre-mapping the distributed block locations across the filesystem, the system can then systematically overwrite each block with random data while the system remains operational, avoiding the need for forced downtime.
2Reliability
If multi-pass overwriting with hashing is used for secure deletion, then data sanitization completeness is improved, but computational overhead and time required increase
Solution Approach 1:
The patent implements configurable sanitization parameters including the number of overwriting passes (e.g., 3, 7, or 19 passes) and the hashing algorithm strength (e.g., SHA-256). Users can adjust these parameters based on their security requirements, allowing flexible trade-offs between sanitization completeness and the time/computational resources consumed.
Solution Approach 2:
The system performs periodic overwriting passes with random data, where each pass writes encrypted random blocks to the target file locations. This periodic action systematically replaces the original data multiple times, with each pass increasing the difficulty of data recovery while the system remains operational.
3Reliability
If random disk rotation is implemented to constantly shuffle data, then data recovery difficulty is improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent implements dynamic random disk rotation that constantly shuffles data blocks across different physical locations on the disk. This dynamic movement makes it extremely difficult to recover deleted data through traditional forensic methods, as the original block locations become unpredictable and change over time while the system operates normally.
Solution Approach 2:
The system introduces an intermediary layer of random data blocks that are written between the original data and the deletion process. These intermediary random blocks act as a barrier, making it difficult to recover original data even if some blocks are accidentally left behind, without requiring complex hardware modifications.
Data Source
AI summary
A system deletes and sanitizes files in a distributed file system. The system also randomizes rotation of data in a distributed file system.