Companion Metadata for Atomic Map-Reduce Appends
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Map-reduce systems using non-POSIX append-only file systems, such as Hadoop's HDFS, face challenges in handling failed append tasks, leading to partially written data that cannot be deleted, resulting in performance and manageability issues due to the inability to delete partially appended files and the overhead of creating numerous small files.
Innovation Solution
The implementation of a management process that maintains companion metadata files to track logical end-of-file (EOF) positions for data files, using global versioning to manage successful and failed append tasks, allowing for atomic incremental loads and safe appends, while avoiding the creation of redundant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If new files are created each time for append operations in an append-only file system, then data can be safely stored without deletion issues, but substantial overhead occurs in a file system designed for large files
Solution Approach 1:
The patent segments the file management into two parts: the actual data file and a companion metadata file. The metadata file tracks the logical EOF and append status, allowing the system to manage append operations efficiently without creating new data files each time. This segmentation enables the system to distinguish between successful and failed append tasks while maintaining file system efficiency.
Solution Approach 2:
The companion metadata file acts as an intermediary between the append operations and the actual data files. It maintains the logical EOF position and tracks append status, serving as a mediator that allows the system to manage file appends efficiently without directly manipulating the data files themselves. This intermediary enables atomic incremental loads while preventing redundant data creation.
2Reliability
If partially appended data is kept in files when tasks fail, then data can be recovered, but the inability to delete these files causes performance and manageability issues
Solution Approach 1:
The companion metadata file provides feedback about the status of each append task (successful or failed) by maintaining the logical EOF position. When an append task fails, the metadata reflects this by not updating the logical EOF, allowing the system to identify and handle failed tasks appropriately without creating management issues from undetected partial writes.
Solution Approach 2:
The patent replaces the mechanical file deletion operation with a logical EOF tracking mechanism. Instead of physically deleting failed append data, the system uses metadata to mark the logical end of valid data, effectively ignoring partial writes without requiring file deletion. This substitution eliminates the need for complex file management operations while maintaining data integrity.
3Ease of manufacture
If multiple small files are created instead of appending to large files, then data organization is simplified, but substantial overhead occurs in the file system
Solution Approach 1:
The companion metadata file serves multiple functions: it tracks the logical EOF, records append status, enables atomic incremental loads, and prevents redundant data creation. This single metadata structure provides universal support for all append operations on the associated data file, eliminating the need for complex file creation and organization logic while maintaining file system performance.
Data Source
AI summary
Augmenting data files in a repository of an append-only file system includes maintaining a companion metadata file for each corresponding data file in a map-reduce system using the append-only file system. Each companion metadata file tracks a logical end-of-file (EOF) for each data file. Global versioning of each companion metadata is maintained. A map-reduce append job is performed for a set of data files using a current global version number for the companion metadata file. The map-reduce job including multiple append tasks. For each successful append job, a logical EOF for each appended file is incremented to a new physical EOF. For each failed append task of the append job, a logical EOF is maintained for each failed append task by not incrementing the logical EOF for each failed append task.