Distributed File System Metadata Update via Row Group Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Append-Only file systems with columnar storage lack an effective method for dynamically updating metadata, resulting in high computing resource overhead and time consumption due to the need for regenerating entire files.
Innovation Solution
A metadata updating method that splits data records into multiple row groups, reorganizes the file format to include actual data blocks, data index blocks, local metadata blocks, metadata index blocks, and file footers, allowing for incremental updates of metadata without regenerating entire files, using the Append-Only file system's appending feature.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata updating is performed in existing Append-Only file systems with columnar storage, then the file system maintains high write performance and consistency, but the metadata update operation requires regeneration and rewriting of all corresponding storage files, resulting in huge computing resource overhead and time consumption
Solution Approach 1:
The patent segments the file structure into distinct components: data blocks, metadata blocks, and footers. This segmentation allows metadata to be updated independently without regenerating entire files. The metadata blocks are further divided into column-specific metadata, enabling granular updates of only the affected metadata portions while preserving the Append-Only property of the file system.
Solution Approach 2:
The patent extracts metadata from the main file structure and stores it in separate metadata blocks. This extraction allows metadata to be manipulated independently from the data blocks. When metadata needs updating, only the extracted metadata blocks are modified and reappended, rather than regenerating the entire file, thus improving update efficiency while maintaining consistency.
2Stability of the object's composition
If metadata updating is performed in existing Append-Only file systems, then the file system structure is maintained, but the operation requires rewriting of all corresponding storage files, leading to huge time consumption
Solution Approach 1:
The patent performs preliminary organization of metadata into separate metadata blocks with clear delimiters and structure markers. This preliminary action establishes a file structure that anticipates future update operations, allowing metadata to be modified by simple append operations rather than requiring complete file rewrites, thus reducing update time while maintaining structural stability.
Solution Approach 2:
The patent introduces dynamic metadata blocks that can be independently modified and reappended to the file. Unlike static file structures that require complete regeneration for any modification, this dynamic approach allows the metadata portion to be updated flexibly by appending only the changed portions, significantly reducing update time while preserving the overall file structure integrity.
3Adaptability or versatility
If metadata updating is performed in existing Append-Only file systems, then the columnar storage structure is preserved, but the operation requires regeneration of entire files, resulting in extremely huge computing resource overhead
Solution Approach 1:
The patent segments the columnar storage structure into independent data blocks and metadata blocks. Each column's data remains in its dedicated data block, preserving the columnar storage capability. The metadata blocks contain column-specific metadata that can be updated independently. This segmentation enables adaptability for columnar operations while reducing computing overhead by avoiding regeneration of unchanged data blocks.
Solution Approach 2:
The patent applies local quality by making only the metadata blocks subject to update operations, while the data blocks remain unchanged. This localized approach preserves the columnar storage structure's integrity and capabilities where needed, while minimizing computing resource overhead by avoiding unnecessary regeneration of the entire file. The update operation is confined to the specific metadata portions that require changes.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The present invention discloses a metadata updating method and apparatus based on columnar storage in a distributed file system, and a host, where the method includes: acquiring to-be-updated metadata in a data table of the distributed file system, splitting data records of the data table into multiple row groups on a row basis, and converting the data table into global file metadata and multiple row group files, where the row group file includes an actual data block, a data index block, a local metadata block, a metadata index block, and a file footer; determining whether the to-be-updated metadata belongs to the global file metadata, and if the to-be-updated metadata does not belong to the global file metadata, then: updating local metadata; and adding an updated local metadata block, an updated metadata index block, and an updated file footer to the multiple row group files according to updated local metadata. According to the method in the present invention, dynamic updating of metadata is implemented, which dramatically saves a time of executing an updating operation of this type and needed computing resources.