Neural Network Linkage Data Generator for Database Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in tracing user data across databases due to low visibility of relationships between data tables, leading to resource-intensive efforts in data discovery, redundancy reduction, and compliance with data privacy regulations, as existing linkage solutions suffer from low performance, scalability, and high costs.
Innovation Solution
A system utilizing a neural network to determine and classify links between data tables based on metadata, enabling efficient data linkage by clustering tables, generating links, and classifying them according to a link classification criterion, which can include machine learning and Siamese neural networks for accurate and scalable data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If granular data linkage solutions are used to provide data-level connections, then data accuracy is improved, but system performance deteriorates and costs increase
Solution Approach 1:
The patent segments the data linkage process into two distinct levels: (1) table-level linkage using metadata and machine learning to identify relationships between data tables, and (2) record-level linkage only when needed. This segmentation allows the system to achieve accurate data connections while avoiding the performance penalty of processing every record pair, as the ML model first filters to relevant table pairs before any granular comparison occurs.
Solution Approach 2:
The system performs preliminary actions by using metadata analysis and machine learning models to pre-identify potential linked tables and columns before actual data comparison. This preliminary filtering step creates a reduced search space, so that when granular record-level linkage is performed, it is done only on a small subset of candidate pairs, thereby maintaining accuracy while improving performance.
2Loss of information
If comprehensive data discovery is performed across all databases, then data relationship visibility is improved, but resource consumption increases
Solution Approach 1:
The patent extracts and utilizes metadata from data tables as a separate, lightweight information layer that contains structural and descriptive information about the data. By performing linkage analysis on this extracted metadata rather than on the actual data records, the system achieves comprehensive data relationship visibility while consuming minimal computational resources, as metadata is typically orders of magnitude smaller than the full dataset.
Solution Approach 2:
The system introduces machine learning models as intermediaries between the raw data and the linkage analysis process. These ML models process metadata and generate linkage predictions, serving as a computational mediator that translates structural data characteristics into relationship insights without requiring exhaustive comparison of actual data records, thereby reducing resource consumption while maintaining visibility.
3Measurement precision
If manual data linkage creation is performed, then linkage accuracy is improved, but time consumption increases
Solution Approach 1:
The system implements self-service by enabling data tables to describe themselves through metadata that contains information about their structure, content, and relationships. The machine learning models automatically process this self-provided metadata to identify linkages, eliminating the need for manual human analysis while maintaining high accuracy through the models' ability to learn from metadata patterns and make intelligent linkage predictions.
4Adaptability or versatility
If data tables are siloed within teams, then data ownership is improved, but data relationship discoverability deteriorates
Solution Approach 1:
The patent implements a universal metadata schema and standardized linkage interface that works across different data teams, schemas, and database types. This universal framework allows each team to maintain ownership of their data tables while the standardized metadata structure enables automated discovery of relationships across team boundaries, as the ML models can process metadata from any team's tables using the same universal interface.
Data Source
AI summary
A system can determine a cluster of tables from a plurality of tables, determine, using a neural network, a link between a pair of columns from respective tables of the cluster of tables, wherein the pair of columns satisfy a relatedness criterion, and classify, using the neural network, the link according to a link classification criterion, wherein the link satisfies the link classification criterion.