Sequential Data Imputation via Denoising Autoencoder
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Electronic health records (EHRs) often contain missing and corrupted data, which hinders their use in medical research and analysis due to their longitudinal and incomplete nature, posing privacy and security concerns that complicate sharing across organizations.
Innovation Solution
A system utilizing a denoising autoencoder and autoregressive recurrent neural network (RNN) to impute missing/corrupted data by iteratively training models to reconstruct and predict sequential data, ensuring outputs are within predetermined tolerance levels, thereby addressing the inefficiencies of existing solutions that handle missing patterns separately from prediction models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If EHR data is shared across organizations for research purposes, then research insights and tool development are enhanced, but privacy and security risks increase
Solution Approach 1:
The patent introduces an imputation system as an intermediary layer between EHR data and research applications. This system fills missing data points using predictive models trained on aggregated patterns, allowing research to be conducted on imputed datasets without exposing raw patient information. The intermediary processing enables data utility while maintaining privacy boundaries.
2Reliability
If missing data points are removed from EHRs, then data privacy is protected, but data completeness and research utility decrease
Solution Approach 1:
The patent creates copies of missing data points through predictive imputation rather than using the original raw data. Multiple imputed values are generated based on learned patterns from similar patient profiles and temporal sequences. These synthesized copies preserve statistical properties and relationships needed for research while being decoupled from identifiable patient information.
Solution Approach 2:
The imputation system performs self-service by automatically filling missing data points using embedded predictive models. The system learns from the data structure and relationships within EHRs, then autonomously generates plausible values for missing entries without requiring manual intervention or exposure of sensitive raw data, enabling continuous data completion while maintaining privacy.
3Measurement precision
If separate models are used for handling missing patterns and prediction, then model specialization is achieved, but system complexity and computational overhead increase
Solution Approach 1:
The patent merges the missing pattern handling and prediction functions into a unified neural network architecture. The model simultaneously learns to identify missing data patterns and generate imputed values through integrated layers that process both tasks together. This consolidation reduces the number of separate models needed while maintaining specialized capabilities through shared learned representations and joint optimization.
Data Source
AI summary
A method and system of imputing corrupted sequential data is provided. A plurality of input data vectors of a sequential data is received. For each input data vector of the sequential data, the input data vector is corrupted. The corrupted input data vector is mapped to a staging hidden layer to create a staging vector. The input data vector is reconstructed based on the staging vector, to provide an output data vector. adjusted parameter of the staging hidden layer is iteratively trained until it is within a predetermined tolerance of a loss function. A next input data vector of the sequential data is predicted based on the staging vector. The predicted next input data vector is stored.