Hierarchical Metadata Scanning for Large-Scale Data Set Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating each file in large data storage systems to identify data sets with shared metadata elements is inefficient and intractable, especially when dealing with millions or billions of files.
Innovation Solution
A data set discovery system that uses parallel processing and heuristic scanning criteria to identify common file metadata elements across nodes, generating unique identifiers for data sets and optimizing the scanning process by leveraging sibling node correlations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each file is evaluated individually to identify data sets with shared metadata elements, then complete data set discovery is achieved, but computational efficiency deteriorates and the task becomes intractable
Solution Approach 1:
The patent segments the data storage system into a hierarchy of nodes (root nodes, intermediate nodes, and leaf nodes). Instead of evaluating every file individually, the system evaluates metadata at the node level, grouping files under common node identifiers. This segmentation reduces the evaluation scope from individual files to representative nodes, maintaining data set discovery capability while dramatically improving computational efficiency.
Solution Approach 2:
The patent introduces node identifiers as intermediary elements between files and data sets. Rather than directly comparing every pair of files, the system uses node identifiers as mediators to infer data set membership. Files sharing common node identifiers are grouped into data sets, eliminating the need for exhaustive pairwise file comparisons and reducing computational complexity.
2Productivity
If parallel processing is used to improve scanning speed, then productivity increases, but system complexity increases
Solution Approach 1:
The patent divides the data storage system into multiple independent node segments that can be processed in parallel. Each node evaluation is independent and can be performed concurrently by different processing units. This segmentation enables straightforward parallelization without requiring complex coordination mechanisms, as each worker process handles a discrete node or set of nodes independently.
Solution Approach 2:
The patent implements a self-service parallel processing model where worker processes autonomously evaluate nodes and generate results without requiring complex inter-process coordination. Each worker process independently retrieves node metadata, performs evaluation, and stores results, eliminating the need for sophisticated task scheduling and result aggregation mechanisms.
Data Source
AI summary
Data set discovery is disclosed, including: identifying first file metadata elements for a first file associated with a node in a hierarchy of data; identifying second file metadata elements for a second file associated with the node; identifying common file metadata elements among the first file metadata elements and the second file metadata elements; and determining that the common file metadata elements represent a data set comprising at least the first file and the second file.