Automated Metadata Asset Creation via Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face challenges in efficiently managing large datasets, leading to issues such as duplicate records, inconsistent responses, and human error in determining record inclusion, which degrade storage efficiency and performance.
Innovation Solution
Utilizing machine learning models, including match, in-database, and out-of-policy models, to calculate probabilities of record matches and compliance with database policies, and a selector model to generate accurate new records from multiple inputs, minimizing duplicates and ensuring policy adherence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual review and matching of database records is performed, then record accuracy can be maintained, but labor cost and time consumption increase significantly
Solution Approach 1:
The patent replaces manual mechanical review processes with automated machine learning models. The match machine learning model automatically compares incoming records against existing database records, substituting human reviewers with an automated system that processes records rapidly while maintaining high accuracy through learned patterns from training data.
Solution Approach 2:
The system enables self-service through automated record matching and duplicate detection. The match machine learning model independently evaluates incoming records, determines duplicates, and suggests actions without requiring manual intervention for each record, allowing the database system to serve itself in the record management process.
2Productivity
If heuristic rules are used to determine record matching, then processing speed improves, but measurement precision deteriorates due to rigidity and inability to detect important differences
Solution Approach 1:
The patent transforms the rigid heuristic rules into flexible machine learning models that can adapt their parameters based on learned patterns. The match machine learning model adjusts its matching criteria dynamically based on the specific characteristics of records being compared, allowing it to capture subtle important differences that fixed rules would miss while maintaining high processing speed.
Solution Approach 2:
The system transitions from static heuristic rules to dynamic machine learning-based matching. The match machine learning model continuously adapts its behavior based on the input data characteristics, enabling it to flexibly adjust matching sensitivity and specificity according to the specific record types being processed, thereby improving both speed and accuracy.
3Reliability
If multiple database records are created to ensure comprehensive information, then data completeness improves, but storage space consumption and processing overhead increase
Solution Approach 1:
The patent extracts and identifies duplicate records using the match machine learning model before they are added to the database. By detecting and removing potential duplicates at the point of ingestion, the system prevents unnecessary storage consumption while ensuring that only unique, valuable records are retained, thus maintaining data completeness without bloat.
Solution Approach 2:
The system applies preliminary anti-action by proactively preventing duplicate records from being inserted into the database in the first place. The match machine learning model evaluates incoming records against existing ones and blocks duplicates before they consume storage space, thereby avoiding the need for later deduplication operations and reducing overall storage requirements.
4Quantity of substance
If strict matching criteria are applied to avoid duplicates, then storage efficiency improves, but false negatives increase causing important information to be omitted
Solution Approach 1:
The patent applies partial matching criteria through the match machine learning model, which evaluates multiple features and attributes of records rather than relying on a single strict criterion. The model can identify duplicates even when records have some differences, by weighing various attributes differently based on their importance, thus avoiding false negatives while maintaining storage efficiency.
Data Source
AI summary
Systems and methods are described that employ machine learning models to optimize database management. Machine learning models may be utilized to decide whether a new database record needs to be created (e.g., to avoid duplicates) and to decide what record to create. For example, candidate database records potentially matching a received database record may be identified in a local database, and a respective probability of each candidate database record matching the received record is output by a match machine learning model. A list of statistical scores is generated based on the respective probabilities and is input to an in-database machine learning model to calculate the probability that the received database record already exists in the local database.