Granular Data Self-Healing via Bitmap Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems, updating large files after a server failure is resource-intensive and degrades performance due to the need to scan entire files to determine differences between replicas, especially for files like virtual machine image files that are 1 Terabyte or larger.
Innovation Solution
The system uses metadata and index files with bitmaps to identify and update specific portions of outdated replicas, allowing granular data self-healing by marking bits in the bitmap to record operations, thereby avoiding the need to scan entire files and update only the necessary parts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system scans entire files to determine differences between replicas, then data integrity is ensured, but resource consumption and update time increase significantly
Solution Approach 1:
The patent divides large files into smaller chunks and uses bitmaps to represent each chunk's state. Instead of scanning entire files, the system only processes and compares specific chunks that may have changed, significantly reducing the time and resources required while maintaining data integrity through selective verification.
Solution Approach 2:
The patent extracts only the necessary portions of file data that need to be updated or verified, rather than processing entire files. By using metadata and bitmaps to identify specific chunks requiring attention, the system removes unnecessary data processing overhead while ensuring integrity where needed.
2Reliability
If the system scans entire files to determine differences between replicas, then complete data verification is achieved, but system performance degrades
Solution Approach 1:
The patent segments large files into manageable chunks represented by bitmaps in metadata. This allows the system to verify and update only specific segments rather than scanning entire files, maintaining verification reliability for changed portions while preserving overall system performance through reduced processing scope.
Solution Approach 2:
The patent applies partial action by performing verification and update operations only on necessary file portions identified through metadata comparison, rather than executing complete file scans. This selective approach maintains data verification effectiveness while preventing performance degradation from unnecessary processing.
3Reliability
If the system updates large files after server failure, then data availability is restored, but resource intensity increases
Solution Approach 1:
The patent extracts and processes only the specific file chunks that require updating after server failure, using metadata bitmaps to identify changed portions. This selective extraction approach restores data availability across replicas while significantly reducing the computational resources and energy required compared to updating entire files.
Solution Approach 2:
The patent divides the update process into manageable file chunks, allowing parallel processing and efficient resource utilization. By segmenting the restoration task, the system recovers data availability across replicas without concentrating excessive resource demands on a single operation.
4Manufacturing precision
If the system processes large files granularly, then update precision is improved, but metadata complexity increases
Solution Approach 1:
The patent creates simplified bitmap representations of file chunks as metadata, which are much smaller and simpler than processing actual file data. These bitmap copies enable precise tracking of chunk states and changes without the complexity of managing full file metadata, achieving high update precision with manageable metadata overhead.
Data Source
AI summary
A first storage server of the file system receives a request to perform an operation on the data file. The operation is to be performed on a first replica of the data file stored at the first storage server and a second replica of the data file stored at a second storage server of the file system. The first storage server configures first metadata associated with a first index file to indicate that the operation is to be performed on a first portion of the first replica. The first storage server determines the second replica is an outdated state indicating that the operation on the second replica has not been performed by the second storage server. In response to the second replica being in the outdated state, updating a first portion of the second replica identified in view of the first metadata and corresponding to the first portion of the first replica.