Dataset Ranking via Graph-Based Lineage and Composite Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data discovery methods in data centers rely on simple text searching, which fail to scale with massive data volumes and lack effective ranking mechanisms, leading to overwhelming results and difficulty in identifying relevant datasets.
Innovation Solution
Implementing a dataset rank metric that generates a graph connecting datasets based on data lineage, using weighted composite scores to iteratively rank datasets, considering data quality, reliability, and popularity, to present the most relevant datasets for a given task.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If simple text searching is used to search datasets, then the search process is simple and fast, but the search results become overwhelming and irrelevant when searching large data centers
Solution Approach 1:
The patent segments the search problem by introducing a graph structure that divides datasets into nodes and their relationships into edges. This segmentation allows the system to process and rank datasets based on their connections rather than treating all datasets uniformly, thereby improving result relevance while maintaining operational simplicity.
Solution Approach 2:
The patent introduces a graph-based intermediary structure between the user's search query and the actual datasets. This intermediary (the graph with nodes and edges) mediates the search process by filtering and ranking datasets based on their relationships, thus improving result relevance without requiring complex search operations from the user.
2Device complexity
If current data discovery methods focus on isolated metrics for individual datasets, then the ranking process is simple, but the results fail to scale with massive data volumes
Solution Approach 1:
The patent merges multiple ranking factors into a unified graph-based approach. Instead of treating individual dataset metrics in isolation, the system combines data lineage, quality, reliability, and popularity into an integrated graph structure, enabling scalable ranking that considers relationships between datasets rather than processing them independently.
Solution Approach 2:
The patent adds a relational dimension to the ranking process by introducing graph edges that represent data lineage and connections between datasets. This dimensional change from isolated metrics to relational structures enables the system to scale effectively by leveraging the graph's hierarchical and interconnected nature rather than processing datasets in isolation.
3Quantity of substance
If thousands of search results are returned for a query, then the search covers comprehensive data, but users cannot identify relevant datasets without effective ranking
Solution Approach 1:
The patent applies local quality by assigning different weights and characteristics to different parts of the graph based on their specific properties (e.g., data quality, reliability, popularity). This allows the system to highlight locally optimal datasets in the search results while maintaining comprehensive coverage, making it easier for users to identify relevant datasets among many results.
Data Source
AI summary
A method implements a dataset rank metric for measuring dataset relevance. Metadata is identified for a plurality of datasets. A graph structure is generated in storage. The graph structure includes a multitude of nodes connected by a multitude of edges. Each node of the multitude of nodes representing a respective dataset of a multitude of datasets, the multitude of edges connecting the multitude of nodes according to a data lineage determined from metadata of the multitude of datasets. A composite score is generated for each node of the graph. The computer processor iteratively processes the composite scores for the multitude of nodes of the graph to generate a dataset rank for each dataset. The multitude of datasets is presented in an interface, sorted according to the respective dataset rank of each dataset.