Neural Network Unit Conversion for Data Lake Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database warehousing systems face challenges in integrating measurements from different sources with varying and often unknown units, requiring metadata and manual tuning for each new data format or transformation algorithm, and lack a mechanism for processing unseen data formats.
Innovation Solution
A method involving a generative model to estimate the true distribution of measurement data and training unit conversion neural networks to convert measurements to a consistent unit, allowing for automatic conversion without requiring known units or metadata, enabling integration of data from diverse sources into a single format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual metadata-based unit conversion is used, then conversion accuracy is maintained for known units, but the system cannot process unseen data formats and requires continuous manual tuning
Solution Approach 1:
The system employs neural networks that automatically learn unit conversion relationships from data patterns without requiring manual metadata or continuous human tuning. The model self-adapts to new data formats by learning from the data itself, enabling autonomous processing of unseen units while reducing operational complexity.
2Quantity of substance
If multiple data sources with different units are integrated, then data comprehensiveness is improved, but the complexity of unit harmonization increases
Solution Approach 1:
The system transforms the unit harmonization problem from a complex manual mapping task into a learnable parameter optimization problem. Neural networks automatically adjust conversion parameters by learning from data patterns, enabling seamless integration of multiple data sources with different units while reducing the complexity of harmonization operations.
3Productivity
If traditional conversion methods are used, then processing speed is adequate for known units, but the number of iterations required for integration increases
Solution Approach 1:
The system performs preliminary learning of unit conversion relationships by training neural networks on available data before actual integration operations. This pre-learning phase enables the model to quickly adapt to new data formats and units, reducing the number of iterations needed during subsequent data integration processes and improving overall productivity.
Data Source
AI summary
Aspects of the invention include techniques for automatically converting measurements from measurement sources having different units. A non-limiting example method includes receiving a plurality of data streams. Each data stream is received from a respective data source and includes measurement data. A target data source is selected from the respective data sources and a generative model is pre-trained on the measurement data of the target data stream to estimate a true distribution of the measurement data in a selected unit of measurement. A unit conversion neural network is trained for each non-target data source to convert the measurement data to the selected unit of measurement. The measurement data of a first non-target data source is converted to the selected unit of measurement using the respective trained unit conversion neural network and the converted measurement data is combined with the measurement data of the target data source in a data lake.