Multi-task LSTM Metadata Classifier for Data Warehouse Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for managing and migrating electronic data warehouses face inefficiencies in metadata labelling, requiring manual intervention, leading to inaccuracies and scalability issues, which can result in security and privacy risks, especially during cloud migration.
Innovation Solution
A computerized metadata classifier using a multi-task machine learning model with LSTM neural networks that automatically predicts and tags metadata labels in real-time, leveraging textual inputs to classify data elements accurately and efficiently, enabling seamless cloud migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual metadata labelling is used, then accuracy can be maintained, but time consumption and resource requirements increase significantly
Solution Approach 1:
The system enables self-service through automated metadata classification where the machine learning model independently performs classification tasks without requiring manual intervention from data stewards, thereby maintaining accuracy while eliminating time consumption associated with manual labelling
Solution Approach 2:
The patent replaces the mechanical manual labelling process with an automated machine learning-based classification system that uses neural networks to automatically assign metadata labels, substituting human cognitive work with computational processes
2Measurement precision
If manual metadata labelling is used, then accuracy can be maintained, but scalability is limited
Solution Approach 1:
The automated machine learning model provides self-service classification that can handle any volume of data without requiring additional manual resources, enabling the system to scale from small to large datasets while maintaining consistent accuracy
Solution Approach 2:
The machine learning model serves multiple functions including classification of various data types, handling different data volumes, and adapting to changing requirements, making the system universally applicable and highly scalable
3Productivity
If automated machine learning classification is implemented, then productivity and scalability improve, but system complexity increases
Solution Approach 1:
The system segments the metadata classification task into distinct components including data ingestion module, machine learning classification engine, and output generation module, making the complex system more manageable and easier to implement
Solution Approach 2:
The patent introduces an intermediary layer in the form of a machine learning model that mediates between raw data and required metadata outputs, simplifying the overall system architecture by encapsulating complexity within the model
4Speed
If metadata classification is performed in real-time, then responsiveness improves, but computational resources required increase
Solution Approach 1:
The system performs preliminary actions by pre-training the machine learning model offline using historical data, so that during real-time classification operations, the model can quickly make predictions without requiring extensive computational resources at execution time
Data Source
AI summary
A computer implemented system and method is provided for predicting and classifying data warehouse metadata labels using deep machine learning. The system includes an input layer for receiving a first and second textual inputs characterizing different aspects of data fields for a data element; an embedding layer embedding the textual inputs separately and independently to a format suitable for long short term memory (LSTM) each provided to a separate LSTM model; a shared layer for concatenating the outputs from each LSTM and applying hard parameter sharing including hidden layers across all tasks; and, a task specific layer classifying the concatenated output into at least one of a possible set of tasks corresponding to separate metadata classifications using a set of simultaneously trained classifiers based on the hard parameter sharing. The multi-task learning model being a single model trained to simultaneously learn multiple classification tasks corresponding to different metadata classifications.