Statistical Join Function for Data Table Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data table join methods are error-prone and result in data expansion when dealing with missing or different data, especially as tables grow in size, making it difficult for organizations to automatically combine large data tables effectively.
Innovation Solution
A statistical join function that splits tables into similar sub-tables based on feature similarity, using statistical measures to determine whether to split or merge tables, and falls back to a normal outer join operation when necessary, to efficiently combine data tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional join methods are used to combine data tables, then the joining process is simple and straightforward, but the data expands mechanically and becomes error-prone when tables contain missing or different data
Solution Approach 1:
The patent changes the approach from mechanical row-by-row joining to statistical parameter-based matching. It calculates statistical parameters (mean, standard deviation, etc.) for columns in both tables and uses these parameters to determine matching rows, thereby preventing mechanical data expansion while maintaining accuracy.
Solution Approach 2:
The patent replaces the mechanical traditional join system with a statistical system. Instead of using standard SQL join operations that mechanically expand data, it employs statistical calculations and probability-based matching to combine tables, eliminating the harmful mechanical expansion effect.
2Productivity
If traditional join methods are used, then the process is straightforward, but it becomes inefficient and error-prone as table size increases
Solution Approach 1:
The patent transforms the join process from examining individual row values to calculating and comparing statistical parameters of entire columns. This parameter-based approach scales efficiently with table size while maintaining or improving matching accuracy through statistical analysis.
Solution Approach 2:
The patent performs preliminary statistical calculations on table columns before the actual joining process. By pre-computing statistical parameters (mean, standard deviation, etc.), it prepares the data in advance, making the subsequent matching process more efficient and accurate, especially for large tables.
3Measurement precision
If statistical parameters are calculated for each column, then matching accuracy improves, but computational complexity increases
Solution Approach 1:
The patent calculates statistical parameters (mean, standard deviation, count) for each column, transforming raw data into summarized statistical representations. This increases measurement precision for matching while managing computational complexity through efficient parameter calculation and comparison.
Solution Approach 2:
The patent calculates statistical parameters for columns that are likely to contain matching information, rather than uniformly processing all columns. This selective approach maintains high precision where needed while reducing unnecessary computational complexity in other areas.
Data Source
AI summary
Aspects of the disclosure relate to joining data tables. A computing platform may input two or more tables into a statistical join function, which may initiate execution of the statistical join function, and where executing the statistical join function comprises applying one or more of: an end condition function, a partition tables function, or an outer join function to generate a new table that includes information from the two or more tables. The computing platform may send, to a user device, the new table and one or more commands directing the user device to display the new table, which may cause the user device to display the new table.