Garbage Data Identification in Distributed Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed object storage systems, abnormal interruptions like power failures can result in tail objects being written to the data storage pool without the corresponding manifest structure, leading to inaccessible 'garbage data' when the head object information is not written.
Innovation Solution
A method and system that identifies garbage data by acquiring a head object identifier, querying the data storage pool for a target data group, marking the corresponding tail object as a target tail object, and marking other tail objects as garbage data, ensuring only correctly associated tail objects are retained.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If tail objects are written into the data storage pool during abnormal interruption, then the writing process continues, but the manifest structure is not written and the tail objects become inaccessible garbage data
Solution Approach 1:
The patent applies preliminary action by writing the head object and manifest structure to the data storage pool before writing tail objects. This ensures that if an abnormal interruption occurs during tail object writing, the head object and manifest structure are already safely stored, allowing for identification and recovery of valid data while marking incomplete tail objects as garbage for later cleanup.
2Reliability
If all tail objects are retained in the data storage pool, then data completeness is maintained, but disk space is wasted on inaccessible garbage data
Solution Approach 1:
The patent applies the taking out principle by extracting and marking garbage tail objects (those without corresponding manifest structures) from the data storage pool. The marking process identifies these objects as garbage data, separating them from valid data, which enables subsequent deletion to free up disk space while preserving data completeness of valid objects.
Solution Approach 2:
The system applies self-service by automatically identifying and marking tail objects as garbage based on the absence of corresponding manifest structures. This automated identification process eliminates the need for manual garbage detection, allowing the system to self-manage data integrity and space optimization through periodic garbage collection operations.
3Ease of manufacture
If the manifest structure is written after tail objects, then the writing process is simpler, but abnormal interruptions result in inaccessible garbage data
Solution Approach 1:
The patent applies preliminary action by reversing the writing order: the head object and manifest structure are written to the data storage pool before the tail objects. This ensures that if an abnormal interruption occurs during tail object writing, the manifest structure is already safely stored, enabling identification of valid versus garbage data through the presence or absence of corresponding manifest entries.
Data Source
AI summary
A method and system for identifying garbage data, an electronic device, and a storage medium. The method includes: uploading an object to a distributed object storage system; acquiring a head object identifier in an index storage pool of the distributed object storage system; querying a data storage pool for a target data group corresponding to the head object identifier; marking a tail object corresponding to a tail object identifier in the target data group as a target tail object; and marking tail objects in the data storage pool other than the target tail object as garbage data.