Cross-Table Similarity Modeling for Structured Data Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing structured database systems face challenges in efficiently and reliably performing predictive data analysis across structured data objects, particularly in detecting similarities and anomalies in table data objects, leading to increased operational and storage loads.
Innovation Solution
Utilizing cross-table data similarity score generation machine learning models, including shared embedding, convolutional, and regression layers, to generate context-aware row-wise representations and predicted similarity scores for table rows, reducing the need for end-user queries and identifying erroneous/anomalous records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional structured database systems are used for predictive data analysis, then data storage and basic query operations are maintained, but operational load increases and similarity detection efficiency decreases
Solution Approach 1:
The system pre-computes and stores embedding representations for table rows during data ingestion or periodic updates, rather than computing similarities on-demand. This preliminary action transforms raw table rows into embedded representations that capture semantic meaning, enabling efficient similarity comparison later without heavy operational load during query time
Solution Approach 2:
The patent replaces traditional mechanical similarity comparison methods (row-by-row comparison algorithms) with a machine learning-based embedding system. By substituting computational mechanics with pre-trained neural network models, the system achieves faster similarity detection while reducing operational complexity
2Measurement precision
If comprehensive data analysis is performed across all table rows, then detection precision improves, but processing time increases
Solution Approach 1:
The system extracts only the most relevant features from table rows by transforming them into fixed-dimensional embedding vectors. This extraction process captures essential semantic information while discarding redundant data, enabling precise anomaly detection with reduced processing requirements and faster execution time
Solution Approach 2:
The patent transforms table row data from its original high-dimensional format into compressed embedding representations with optimized dimensionality. This parameter transformation maintains detection precision by preserving semantic relationships while reducing the computational complexity and processing time required for analysis
3Extent of automation
If machine learning models are deployed for similarity detection, then automated anomaly detection improves, but system complexity increases
Solution Approach 1:
The patent employs a universal embedding model that serves multiple functions: similarity detection, anomaly detection, and data classification. This multi-functional approach automates various analytical tasks through a single model architecture, reducing the need for multiple specialized models and simplifying system complexity while maintaining high automation capability
Data Source
AI summary
Various embodiments of the present invention provide methods, apparatus, systems, computing devices, computing entities, and/or the like for performing predictive data analysis with respect to structured data objects. Certain embodiments of the present invention utilize systems, methods, and computer program products that perform predictive data analysis with respect to structured data objects by utilizing at least one of cross-table data similarity score generation machine learning models and unsupervised anomalous table row detection machine learning models.