Distributed Storage Metadata Generation via Split Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed data processing systems, such as Hadoop, face inefficiencies in managing metadata for large datasets due to the need for duplicating and copying data during annotation processes, leading to excessive computational costs and data duplication.
Innovation Solution
A storage layer for distributed storage systems that allows for the creation and access of annotation data layers as separate files, using a hybrid file format as a thin wrapper layer on a distributed file system, enabling independent writer tasks and metadata generation without duplicating data, by writing data in portions and recording split boundaries to generate partial metadata files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is duplicated and copied during annotation processes in current distributed systems, then annotation data can be added to original datasets, but computational costs increase and excessive data duplication occurs
Solution Approach 1:
The patent segments the dataset into multiple splits distributed across different machines, with each machine processing only its local split for annotation. This eliminates the need to duplicate and copy entire datasets, as annotations are performed in-place on each split. The segmentation principle directly resolves the contradiction by enabling annotation capability while avoiding the computational overhead of data duplication.
Solution Approach 2:
The patent extracts only the necessary metadata (split boundaries, file paths, row counts) from each distributed split rather than copying the actual data. This extraction approach allows the system to track and manage annotated data across distributed machines without incurring the computational cost of duplicating large datasets, thus resolving the contradiction between annotation capability and computational efficiency.
2Reliability
If entire datasets are read and processed for annotation in current systems, then complete data can be annotated, but processing time and computational resources are excessive
Solution Approach 1:
The patent divides the dataset into multiple splits distributed across different machines, allowing parallel annotation processing on each split. This segmentation enables complete annotation of the entire dataset while significantly reducing processing time through distributed parallel execution, directly resolving the contradiction between annotation completeness and processing time.
Solution Approach 2:
Each machine performs annotation on only its local split portion rather than processing the entire dataset. This partial action approach, when aggregated across all machines, achieves complete dataset annotation with reduced processing time and computational resources, resolving the contradiction between reliability and time loss.
3Adaptability or versatility
If data is copied multiple times for multiple annotation operations, then multiple annotations can be added, but data duplication increases exponentially
Solution Approach 1:
The patent maintains data segmentation into splits across distributed machines, allowing multiple annotation operations to be performed in-place on each split without copying data. This approach enables unlimited annotation layers while preventing exponential data duplication, as each annotation modifies only the metadata of its local split rather than creating new data copies.
Solution Approach 2:
The patent extracts and stores only metadata (split boundaries, file paths, row counts) rather than copying actual data for each annotation operation. This extraction mechanism enables multiple annotations to be tracked and managed without increasing data duplication, directly resolving the contradiction between multiple annotation capability and data quantity.
4Productivity
If distributed writer tasks are used for writing data, then writing can be parallelized, but metadata generation becomes complex and difficult
Solution Approach 1:
The patent segments metadata generation into independent tasks corresponding to each distributed writer task. Each writer task generates metadata only for its local split, and these partial metadata files are later aggregated. This segmentation of metadata generation mirrors the data segmentation, maintaining parallel writing speed while simplifying metadata management through modular, distributed processing.
Solution Approach 2:
Writer tasks generate partial metadata files during the writing process itself, before the final aggregation step. This preliminary action approach allows parallel writing to proceed independently while metadata is being prepared in advance, reducing the complexity of coordinated metadata generation and enabling straightforward aggregation of partial metadata files into a complete metadata file.
Data Source
AI summary
Disclosed are embodiments for generating a dataset metadata file based on partial metadata files. In one embodiment, a method is disclosed comprising receiving data to write to disk, the data comprising a subset of a dataset; writing a first portion of the data to disk; detecting a split boundary after writing the first portion; recording metadata describing the split boundary; continuing to write a remaining portion of the data to disk; and after completing the writing of the data to disk: generating a partial metadata file for the data, the partial metadata file including the split boundary, and transmitting the partial metadata to a partial metadata collector.