Automated Label Linking via LSH Forest and ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual linking of technical metadata to domain definitions in cloud computing environments is a time-consuming and labor-intensive process, especially when dealing with large volumes of data, making it difficult for data producers to find and manage trusted data effectively.
Innovation Solution
A method using a machine-learning model and Locality Sensitive Hashing (LSH) forest to automatically recommend similar labels by generating feature sets for pairs of technical and domain terms, allowing for automated association and improved search capabilities within a cloud-computing data catalog.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual linking of domain definitions to technical metadata is performed, then data governance and data discovery are improved, but time consumption and labor intensity increase significantly
Solution Approach 1:
The patent replaces the manual mechanical process of linking domain definitions to technical metadata with an automated machine learning system. The system uses trained models to automatically perform the linking that previously required human data stewards to manually search and associate terms, thereby eliminating the time-consuming manual effort while maintaining or improving data governance quality.
Solution Approach 2:
The system enables self-service automation where the data catalog automatically performs the linking operation without requiring human intervention. The machine learning models autonomously analyze technical metadata, generate candidate domain definitions, and establish associations, allowing the system to serve itself rather than relying on continuous manual input from data practitioners.
2Ease of operation
If manual search and linking is performed to improve data discovery, then data understanding is enhanced, but scalability deteriorates when metadata volume grows
Solution Approach 1:
The patent substitutes manual search and linking operations with automated machine learning-based systems. Instead of data practitioners manually searching through growing metadata volumes, the system automatically discovers relationships between technical metadata and domain definitions using trained models, maintaining ease of data discovery regardless of metadata scale.
Solution Approach 2:
The system changes the operational parameters from manual human search behavior to automated algorithmic processing. By transforming the data discovery process from human-cognitive operations to machine-executed algorithms, the system achieves scalability that manual processes cannot match, as computational operations can handle large volumes efficiently without proportional increases in time or effort.
3Productivity
If automated machine learning linking is implemented, then time efficiency and scalability are improved, but system complexity increases
Solution Approach 1:
The patent applies preliminary action by training machine learning models in advance on curated datasets before deployment. The models are pre-trained to understand the relationships between domain definitions and technical metadata, so that during actual operation, the system can quickly perform linking without requiring complex real-time processing. This upfront preparation reduces the operational complexity of the automated system.
Solution Approach 2:
The system introduces trained machine learning models as intermediaries between the raw metadata and the linking operation. Rather than implementing complex rule-based systems or requiring direct human intervention, the patent uses pre-trained models as intermediary components that automatically perform the matching, simplifying the overall system architecture while maintaining high productivity.
Data Source
AI summary
A machine-learning model may be previously trained with a supervised learning algorithm to identify whether a pair of labels provided as input are similar. A locality sensitive hashing forest (LSH) may be generated for the set of candidate labels. When a user later identifies an input label (e.g., by search query, by interface selection, etc.) the input label may be used to query the LSH forest to identify a subset of the candidate labels. This subset may be used to generate respective pairs comprising the input label, one of the subset candidate labels, and a corresponding feature set generated for the pair. This data may be provided to the model to identify a degree to which the pair of labels are similar. The user may be provided one or more recommendations including similar terms identified from the model's output.


