Metadata Clustering for Partial Integration of Disparate Data Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data integration systems face challenges in integrating disparate data sources due to varying naming conventions, different data types, and manual, incomplete, and costly integration processes, often requiring wholesale system conversion or significant investments.
Innovation Solution
A metadata processor utilizing unsupervised and supervised machine learning techniques to cluster metadata from different sources, allowing for automated and partial integration without disrupting existing systems, and incorporating industry standards into the supervised learning process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual integration processes are used to integrate disparate data sources, then integration accuracy can be improved through human review, but the process becomes expensive, time-consuming, and incomplete
Solution Approach 1:
The system performs preliminary automated clustering of metadata using unsupervised machine learning before final integration decisions are made. This pre-processing groups similar data sources together, enabling faster subsequent processing while maintaining accuracy through the two-stage approach of automated clustering followed by selective human review only when needed.
Solution Approach 2:
The patent introduces machine learning models as an intermediary between raw data sources and the integration process. These models automatically analyze metadata patterns, cluster similar sources, and prioritize integration tasks, reducing the need for manual intervention while maintaining high accuracy through the intermediary's intelligent processing.
2Stability of the object's composition
If existing integration reference systems are used to manage data integration, then system stability is maintained, but it becomes difficult to implement new and more effective integration processes
Solution Approach 1:
The system implements dynamic integration processes that can adapt to new data sources and integration requirements. The machine learning models are trained on existing integration references to maintain stability, but can continuously learn and adapt to new patterns, enabling the system to evolve without disrupting existing stable integrations.
Solution Approach 2:
The integration system is segmented into independent modular components: metadata extraction, unsupervised clustering, supervised learning model training, and integration execution. This segmentation allows new integration processes to be implemented as separate modules without disrupting the stability of existing integrated systems.
3Adaptability or versatility
If wholesale system conversion is performed to integrate different data sources, then complete integration is achieved, but transition costs and disruption increase significantly
Solution Approach 1:
The system performs partial integration by clustering and integrating only the most similar and valuable data sources first, rather than attempting to integrate all sources simultaneously. This partial action approach achieves meaningful integration results with lower transition costs and less disruption, while leaving room for gradual expansion to additional sources.
Solution Approach 2:
The patent creates virtual copies and representations of data source metadata that can be analyzed and clustered without moving or converting the actual data sources. This copying approach enables integration planning and execution with minimal disruption to existing systems and lower transition costs.
4Productivity
If automated machine learning processes are used to cluster metadata, then manual effort is reduced, but the complexity of the system increases
Solution Approach 1:
The system implements self-service through automated machine learning processes that perform metadata clustering and integration prioritization without requiring manual configuration or intervention. The unsupervised learning models automatically discover patterns in metadata, and the supervised models learn optimal integration strategies, reducing manual effort while managing complexity through automation.
Data Source
AI summary
A system, device and method are provided for assessing actions of authenticated persons within an enterprise system. The illustrative method includes extracting metadata comprising a plurality of categories from a plurality of data sources, and applying an unsupervised machine learning process to the extracted metadata. A plurality of clusters of the plurality of categories of the extracted metadata is generated, and thereafter one or more review criteria are applied thereto to generate curated clusters. The method includes training a supervised machine learning model with the curated clusters. The method includes, in response to receiving a new metadata input, processing the new metadata input with the trained supervised machine learning model. Data associated with the new metadata input is ingested based on respective clusters output by the trained supervised machine learning model for categories of the new metadata input.