HDFS DataNode Delayed Deletion for Accidental File Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In Hadoop systems, files deleted by misoperation cannot be recovered due to immediate deletion by DataNodes, leading to reduced data security as the Trash mechanism in the NameNode is insufficient in preventing accidental file loss.
Innovation Solution
Implementing a method where DataNodes receive deleting instructions with data block identifiers, suspend deletion until a condition is met, and store these identifiers in a delay queue, allowing for potential recovery before actual deletion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the DataNode immediately deletes data blocks upon receiving a deleting instruction from the NameNode, then the deletion operation is efficient and fast, but the data security is compromised and accidental deletions cannot be recovered
Solution Approach 1:
The patent introduces a Trash directory that temporarily stores deleted files before permanent deletion. When a file is deleted, it is first moved to the Trash directory rather than being immediately removed. This preliminary action allows users to recover files within the retention period while still enabling efficient deletion operations. The NameNode maintains metadata for files in Trash, and only after the retention period expires are the actual data blocks deleted from DataNodes.
Solution Approach 2:
The Trash directory serves as an intermediary between the source directory and permanent deletion. Instead of directly deleting files from their original location, the system uses Trash as a intermediate storage location. This intermediary mechanism preserves data temporarily while allowing the deletion workflow to proceed, enabling both efficient operation and data recovery capability.
2Reliability
If the NameNode maintains metadata for deleted files in the Trash, then file recovery is enabled, but the system complexity increases due to additional metadata management requirements
Solution Approach 1:
The patent merges the Trash directory management into the existing HDFS namespace structure. Rather than creating a separate parallel system for tracking deleted files, the Trash directory is integrated as a regular directory within the file system hierarchy. The NameNode uses the same metadata structures (inode, directory tree) to manage both active and deleted files, reducing overall system complexity while enabling recovery functionality.
3Reliability
If the Trash retention period is extended to allow more time for recovery, then data security is improved, but the storage space consumption increases due to retained deleted files
Solution Approach 1:
The patent implements a configurable retention period for the Trash directory that can be dynamically adjusted based on organizational needs. The retention policy is not fixed but can be modified to balance between recovery requirements and storage constraints. Additionally, the system automatically manages the Trash directory by expiring old entries and freeing storage space once the retention period is exceeded, allowing flexible adaptation to different storage capacity scenarios.
Data Source
AI summary
A method, a device, a node and a system for managing file in distributed data warehouse are provided. The method includes: acquiring, by a data node, a deleting instruction carrying a data block identifier, wherein the deleting instruction is sent by a management node; suspending, by the data node, the deleting instruction; and deleting, by the data node, a data block corresponding to the data block identifier after a condition is met, thereby resolving the technical issue that an accidentally deleted file can not be recovered by setting a trash in the management node in some cases and ensuring the data security of the Hadoop system.