Schema Mapping for Canonical Data Conversion and Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Lenders face challenges in managing diverse data sources with unique schemas that are incompatible with their own SQL databases, leading to a tedious and error-prone manual reorganization process, which is costly and time-consuming due to frequent regulatory and investor information requests.
Innovation Solution
A system and method for converting unique customer data schemas into uniform data schemas using a smart mapper system that includes a mapper module, GUI, data category library, and data migration module, utilizing AI and pattern recognition to automatically match and modify data fields, and an anomaly detection system to identify and flag inconsistent data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual data reorganization is performed to convert unique customer schemas to uniform schemas, then data compatibility is improved, but time consumption and error rate increase
Solution Approach 1:
The patent replaces manual mechanical data reorganization with an automated computer-based system that uses machine learning models and algorithms to perform schema conversion, thereby eliminating the time-consuming and error-prone manual process while maintaining data compatibility
Solution Approach 2:
The patent introduces an intermediary automated mapping system that acts as a mediator between unique customer schemas and uniform target schemas, using predefined mapping rules and machine learning to automatically translate and reconcile different data structures without direct manual intervention
2Adaptability or versatility
If manual data reorganization is performed to convert unique customer schemas to uniform schemas, then data compatibility is improved, but labor cost increases
Solution Approach 1:
The patent implements a self-service automated system that performs schema conversion independently without requiring human labor, using machine learning models trained on schema mapping tasks to autonomously handle the transformation process, thereby eliminating ongoing labor costs while maintaining compatibility
3Productivity
If automated schema conversion is implemented, then productivity is improved, but system complexity increases
Solution Approach 1:
The patent segments the automated schema conversion system into distinct functional modules including data ingestion, schema analysis, mapping rule application, and output generation, allowing each component to be independently developed, maintained, and optimized while collectively achieving high conversion efficiency
Data Source
AI summary
A system for converting a source data feed schema into a canonical data product including a memory for storing computer-executable instructions and a processor for executing the instructions stored on the memory. Execution of the instructions programs the processor to perform operations that include receiving a source data feed having a source schema, identifying a plurality of data fields of the source schema, assigning a data category from a plurality of predefined data categories to each data field of the plurality of data fields, modifying the source schema based on predefined parameters, wherein the source schema is modified to match a target schema, comparing the modified source schema to the target schema, and in response to a determination that the modified source schema matches the target schema, converting the source data feed to a canonical data product having the target schema based on the assigned data categories.