Multi-Source Data Imputation via Mapping Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models face accuracy issues when dealing with incomplete data sets, as missing data can lead to inaccurate or misleading predictions, particularly when data from multiple sources is not available, and existing methods like filling with zeros or averaging can result in unrealistic or inaccurate imputations.
Innovation Solution
A data imputation system that uses multiple mapping models to map data from different sources to a multi-dimensional space, identifying similar entities to impute missing data based on actual values from these entities, ensuring the imputed data is realistic and representative.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is filled in with zeros or placeholder values, then the data set becomes complete, but the accuracy and realism of the data deteriorates
Solution Approach 1:
The patent introduces mapping models as intermediary components that transform data from multiple sources into a unified multi-dimensional space. These mapping models serve as mediators between incomplete data sets and the requirement for complete, accurate data, enabling realistic imputation by finding corresponding entities in the mapped space rather than using simple placeholder values
Solution Approach 2:
The patent creates copies of data patterns from similar entities in the mapped space to impute missing values. Instead of using artificial placeholders, the system copies actual data patterns from corresponding entities that have complete data, thereby maintaining realism and accuracy in the imputed values
2Reliability
If data is extrapolated using averaging techniques, then missing values are filled, but the realism and representativeness of the data deteriorates
Solution Approach 1:
The patent applies local quality by treating each data point and entity individually rather than applying uniform averaging across all data. The mapping models enable entity-specific imputation by finding and copying patterns from locally similar entities in the mapped space, preserving the unique characteristics and representativeness of each data point
3Measurement precision
If data from multiple sources is used, then the accuracy of predictions improves, but the complexity of data processing increases
Solution Approach 1:
The patent merges multiple data sources by mapping their respective data into a unified multi-dimensional space. This consolidation approach allows the system to leverage data from multiple sources for improved prediction accuracy while managing complexity through a unified processing framework rather than handling each source separately
4Productivity
If incomplete data sets are used directly, then processing time is reduced, but prediction accuracy deteriorates
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing mapping models that represent the relationships between different data sources. These pre-computed mappings enable rapid imputation of missing data during prediction, achieving both fast processing and high accuracy without requiring extensive real-time computation
Data Source
AI summary
Methods and systems are presented for imputing missing data items within a first dataset based on data associated with a second dataset that is the nearest neighbor of the first dataset. A first mapping model is configured to map data subsets corresponding to a first data source to first positions in a multi-dimensional space. A second mapping model is configured to map data subsets corresponding to a second data source to second positions in the multi-dimensional space. The first and second mapping models are trained together to reduce a distance between positions mapped by the first and second mapping models based on corresponding data subsets that belong to the same entity. A nearest neighbor dataset to the first dataset is identified based on the first and second mapping models. Data associated with the nearest neighbor dataset is used to impute the missing data items of the first dataset.