Reference Database Canonicalization for Multi-Source Query Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for querying multiple target databases face challenges due to inconsistencies and errors in input data and database entries, leading to complex rule definitions and increased processing complexity, especially when dealing with diverse and dynamic data sources.
Innovation Solution
The proposed method involves using a static, cleansed reference database to validate and standardize input data through reference-based mapping rules, followed by transformation into a canonical form for simplified querying of target databases using target-based query rules, reducing the number and complexity of rules needed from M×N to (M×C)+N.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If direct mapping rules are defined between multiple input sources and target databases, then query functionality is achieved, but the number and complexity of rules increases to M×N
Solution Approach 1:
The patent introduces a reference database as an intermediary between input sources and target databases. Instead of directly mapping M input sources to N target databases (requiring M×N rules), the system maps input sources to the reference database (C rules) and then from the reference database to target databases (N rules), reducing total rules to (M×C)+N where C is the number of reference database tables.
Solution Approach 2:
The patent segments the direct mapping path into two separate mapping paths: (1) input sources to reference database, and (2) reference database to target databases. This segmentation allows each mapping relationship to be simpler and more standardized, reducing overall complexity.
2Loss of information
If direct mapping between diverse input data and target database entries is performed, then information extraction is achieved, but processing complexity increases due to inconsistencies and errors
Solution Approach 1:
The patent performs preliminary validation and standardization of input data against the reference database before querying target databases. By pre-validating input data format, type, and content against the standardized reference database schema, the system reduces processing complexity during actual queries and improves information extraction reliability.
Solution Approach 2:
The reference database serves as an intermediary that standardizes data representation. It provides a canonical format that mediates between diverse input data formats and varying target database schemas, reducing processing complexity by eliminating the need to handle all possible format variations in each direct mapping.
3Reliability
If input data is validated against a reference database, then data quality and match accuracy improve, but additional processing steps are required
Solution Approach 1:
The validation against the reference database is performed as a preliminary step before actual target database queries. While this adds processing time, it prevents wasted queries on malformed or inconsistent data, improving overall system efficiency by filtering out invalid inputs early.
Solution Approach 2:
The system changes the state of input data from unvalidated to validated through comparison with the reference database. This parameter change (validation status) enables more accurate matching by ensuring input data conforms to expected formats and contains required information before querying target databases.
Data Source
AI summary
Input data queries directed at a plurality of target databases and originating from any of a plurality of sources are first converted to validated canonical forms, which are then used to query the target databases. Specifically, upon receiving an input data query, a relatively accurate reference database is selected based on the type of the input data. This reference is then queried for the input data with the intent of finding an exact matching record or a near-matching record that can be considered an exact match and thereby validating the input data. Otherwise, the requesting source is instructed to provide a new query. Once having a validated record, it is converted to a canonical form, which is then used to query the target databases intended to be searched. In a further embodiment, multiple reference databases are queried to determine a canonical form of the data or to determine multiple canonical forms of the data.