Filesystem Metadata Partitioning for Faster ML Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face inefficiencies in filtering and reading large-scale filesystem metadata for machine learning processes, particularly in managing backups for numerous clients, leading to time-intensive operations.
Innovation Solution
The system organizes filesystem metadata in separate partitions based on customers and time ranges, storing it in a column-based format like Parquet, allowing parallel reading and querying specific columns without reading entire files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If filesystem metadata is stored in a traditional flat format, then storage is simple, but reading and filtering large-scale metadata becomes time-intensive
Solution Approach 1:
The patent divides the filesystem metadata into separate partitions organized by customer and time range. Each partition contains only the metadata relevant to specific customers and time periods, enabling selective reading without processing the entire metadata dataset. This segmentation directly addresses the technical contradiction by improving retrieval speed through targeted access while maintaining manageable structure through hierarchical organization.
Solution Approach 2:
The patent implements column-based storage format where different columns store different types of metadata information (e.g., file paths, timestamps, sizes). This allows the system to read only specific columns needed for particular queries rather than reading the complete metadata records. The local quality optimization enables efficient filtering and retrieval operations, resolving the contradiction between productivity and structural complexity.
2Quantity of substance
If the system stores metadata for numerous clients with large datasets, then comprehensive data availability is improved, but processing time increases
Solution Approach 1:
The patent organizes metadata into partitions based on customer identifiers and time ranges. When a query is made, the system can quickly identify which partitions contain the required data and read only those partitions rather than scanning through all client metadata. This segmentation enables the system to handle large volumes of metadata efficiently, reducing processing time while maintaining comprehensive data availability across numerous clients.
Solution Approach 2:
The patent extracts and separates metadata by customer and time range into distinct partitions. This extraction allows the system to isolate and process only the relevant subset of data needed for each query, rather than processing the entire large dataset. The extraction principle directly reduces processing time while preserving the ability to access comprehensive metadata when needed.
3Productivity
If the system reads entire metadata files, then data completeness is ensured, but read efficiency decreases
Solution Approach 1:
The patent stores metadata in column-based format where each column represents a specific attribute (file path, timestamp, size, etc.). When reading metadata, the system can specify which columns are needed and read only those columns rather than reading the entire record. This local quality approach maintains data completeness for the requested attributes while significantly improving read efficiency by minimizing the amount of data transferred and processed.
Data Source
AI summary
Methods, systems, and devices for data management are described. A data management system (DMS) may be employed to manage data associated with one or more computing systems. The DMS may generate metadata when capturing snapshots, which may contain information corresponding to files that have been created, deleted, modified, or otherwise changed since a prior snapshot. The DMS may use learning techniques and experimentation to perform data engineering to analyze metadata such as to perform anomaly detection. The DMS may implement techniques for efficiently storing and organizing metadata for use with machine learning processes—e.g., to support efficiently extracting datasets useful for machine learning experimentation. To obtain a dataset, the DMS may read metadata from different partitions of a storage environment in parallel, and accordingly, metadata may be stored in separate partitions based on customers and time ranges (e.g., to support such parallel reading).