Dataset Joining via Numerical Representation Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for joining datasets from different platforms face challenges due to incompatible data formats, differing column headings, and inconsistent data entries, leading to errors and complexities in data conversion and merging processes.
Innovation Solution
A computer-implemented method that forms string representations of dataset parts, determines numerical representations using techniques like term frequency-inverse document frequency, and applies a random forest classifier to establish mappings between datasets, enabling effective joining and merging of tables based on similarity analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional utility or import functions are used to convert data between platforms, then data compatibility can be achieved, but the process is subject to errors and failures due to opaque and proprietary internal details of platforms
Solution Approach 1:
The patent introduces an intermediary layer consisting of abstract data type definitions and a generic converter framework. This intermediary translates between different platform-specific data formats and a common internal representation, eliminating the need to directly interact with opaque platform internals. The intermediary handles format translation, validation, and error management centrally, improving reliability while reducing complexity at the application level.
Solution Approach 2:
The patent changes the parameters of data representation by defining a standardized set of abstract data types with explicit type signatures. Instead of relying on platform-specific internal representations, the system uses parameterized type definitions that can be consistently translated between different platforms. This parameterization approach enables reliable conversion without requiring deep knowledge of proprietary platform details.
2Adaptability or versatility
If datasets from different platforms are joined using traditional methods, then data merging can be performed, but format and annotation differences cause errors and complexities
Solution Approach 1:
The patent employs abstract data type definitions as intermediaries that serve as a common language between different platforms. These definitions include standardized field names, data types, and validation rules that mediate the joining process. The system translates datasets from various platforms into this common abstract representation before joining, ensuring accurate matching and merging while handling format and annotation differences systematically.
Solution Approach 2:
The patent creates a universal data representation framework that can handle multiple platform formats through a single set of abstract data types. This universal model serves multiple functions: it defines the common schema for joining, provides validation logic, and enables translation from different source formats. The universality of this framework improves both adaptability to different platforms and reliability of joining operations.
3Reliability
If manual data conversion and joining processes are used, then data can be integrated, but the process is time-consuming and labor-intensive
Solution Approach 1:
The patent implements self-service through automated translation and validation functions that operate based on the abstract data type definitions. The system automatically translates data formats, validates constraints, and performs joining operations without manual intervention. The generic converter framework handles format conversion autonomously by applying predefined type mappings, significantly reducing processing time while maintaining the reliability ensured by the structured approach.
Solution Approach 2:
The patent enables continuous automated processing where data translation, validation, and joining operations execute seamlessly in sequence. The abstract data type framework provides continuous guidance through the entire pipeline, from format translation to constraint validation to final joining. This continuous automated action eliminates interruptions and manual steps, reducing overall processing time while maintaining high reliability through systematic validation at each stage.
Data Source
AI summary
A computer-implemented method includes receiving a first dataset including a first table having a first number of parts, where each part of the first number of parts represents either a row or a column of the first table and receiving a second dataset including a second table having a second number of parts, where each part of the second number of parts represents either a row or a column of the second table. For each part of the first number of parts the method includes forming a string representation of the part from a number of values associated with the part and determining a numerical representation of the string representation of the part. For each part of the second number of parts, the method includes forming a string representation of the part from a number of values associated with the part and determining a numerical representation of the string representation of the part. The method includes determining a mapping between at least some parts of the first table and second table including determining that a first set of one or more parts of the first number of parts correspond a second set of one or more parts of the second number of parts based at least in part on a similarity between numerical representations of the parts.