Dataset Merging via Type-Specific Similarity Matrices

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Users face challenges in efficiently merging data from one source into another, especially when the data is categorized differently, leading to inefficiencies and potential errors due to manual handling.

Innovation Solution

A method that involves obtaining existing and new datasets, grouping them by data types, generating similarity features, creating similarity matrices, and using optimization to identify mappings between the datasets, while allowing for user preferences and validation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data from different sources with different categorical arrangements is merged manually, then the merging process can be completed, but the process is time-consuming and error-prone

Engineering Contradiction:

Improvedata merging efficiencyVSAvoidtime for manual data handling

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs self-service by automatically identifying mappings between data sources through algorithmic comparison of column names, data types, and sample values. The automated matching engine independently determines correspondences without manual intervention, resolving the contradiction by enabling high-speed automated merging while eliminating time-consuming manual operations.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical data merging operations with an automated computational system. The system uses algorithms to compare data structures, generate similarity scores, and determine mappings automatically, substituting human manual work with machine-based processing that is both faster and more consistent.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If automated data merging is implemented without user input, then processing speed increases, but accuracy may decrease due to inability to handle nuanced data differences

Engineering Contradiction:

Improveautomated processing speedVSAvoidmapping accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system incorporates feedback mechanisms where user preferences and corrections are fed back into the mapping process. The system generates initial automated mappings, receives user feedback on accuracy, and uses this feedback to refine and adjust mappings, thereby maintaining high processing speed while improving mapping precision through iterative refinement.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The mapping system is dynamic and adaptable rather than static. It adjusts mapping decisions based on data characteristics, similarity scores, and user preferences. The system can dynamically modify mapping configurations to balance automated efficiency with accuracy requirements for different data scenarios.

Inventive Principle:

Principle #15Dynamics

3Measurement precision

If detailed validation and user preference collection is performed for all data mappings, then accuracy improves, but the complexity of the merging process increases

Engineering Contradiction:

Improvemapping validation accuracyVSAvoidmerging process complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The validation process is segmented into distinct stages: automated initial validation, selective detailed validation based on similarity scores and data types, and user preference collection only for ambiguous cases. This segmentation reduces overall process complexity by applying comprehensive validation only where necessary while maintaining high accuracy for critical mappings.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies different levels of validation quality to different data mappings based on their characteristics. High-similarity mappings receive standard automated validation, while low-similarity or ambiguous mappings receive enhanced validation with user preference collection. This local differentiation of validation quality maintains accuracy for critical cases while reducing overall process complexity.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS12204873B1Systems and methods for merging datasets

Publication Date: 2025.01.21 ANAPLAN INC

AI summary

A method for merging data includes: obtaining an existing dataset and a new dataset; grouping the new dataset and the existing dataset based on data types to obtain a plurality of data type groups; performing data type specific processing to generate similarity features; generating a plurality of similarity matrices using the similarity features; performing a comparison between the existing dataset and the new dataset using optimization and the plurality of similarity matrices to identify mappings between the existing dataset and the new dataset; performing data type specific validation to obtain an updated existing dataset, an updated new dataset, matches, partial matches, and new items; obtaining user preferences associated with the mappings, the matches, the partial matches, and the new items; and generating merged data using the updated existing dataset, the updated new dataset, the mappings, the matches, the partial matches, the new items, and the user preferences.