Spectral Data Scaling for ML Noise Removal Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing noise removal methods in spectral data analysis, such as Raman spectroscopy, are ineffective in controlling the degree of noise removal without retraining the machine learning model, potentially damaging high-frequency features when scaling is inappropriate.
Innovation Solution
Adjusting the scaling of spectral data relative to training data allows users to control noise removal without retraining the model, using normalization techniques like min-max, max-value, or Z-score normalization, and applying a machine learning model like a neural network to process the data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If spectral data is normalised to the same scale as training data, then the machine learning model can effectively remove noise, but high-frequency features may be damaged
Solution Approach 1:
The patent applies parameter changes by introducing a scaling factor (parameter) that modifies the normalisation scale of spectral data relative to training data. By adjusting this scaling factor, the system can control the degree of noise removal while preserving high-frequency features, resolving the contradiction between noise removal effectiveness and feature preservation.
2Measurement precision
If spectral data is normalised to higher values, then noise removal effect is reduced, but damage to real high-frequency features is minimized
Solution Approach 1:
The patent uses parameter changes by introducing a scaling factor that can be adjusted to control the normalisation scale. When spectral data is normalised to higher values (using a scaling factor greater than 1), the system minimizes damage to high-frequency features while still maintaining effective noise removal, allowing users to balance these competing requirements.
3Reliability
If spectral data is normalised to lower values, then noise removal effect is enhanced, but damage to real high-frequency features increases
Solution Approach 1:
The patent applies parameter changes by introducing a scaling factor that can be adjusted to control the normalisation scale. When spectral data is normalised to lower values (using a scaling factor less than 1), the system enhances noise removal effectiveness while minimizing damage to high-frequency features, allowing users to optimize the balance between these two objectives.
4Adaptability or versatility
If a new machine learning model is trained with rescaled training data, then noise removal can be controlled, but time consumption increases
Solution Approach 1:
The patent applies preliminary action by pre-training the machine learning model with normalised training data and then enabling users to control noise removal by simply adjusting the scaling factor during application. This eliminates the need to retrain the model for each different noise removal level, significantly reducing time consumption while maintaining adaptability.
Solution Approach 2:
The patent uses parameter changes by introducing a scaling factor that can be adjusted without retraining the model. Users can control the degree of noise removal by modifying this parameter in the data processing stage, rather than requiring time-consuming model retraining, thus resolving the contradiction between adaptability and time loss.
Data Source
AI summary
A method for removing noise from spectral data recorded using a spectrometer. The method includes normalising spectral data to generate normalised spectral data and applying a machine learning model to the normalised spectral data. The machine learning model is trained to remove noise from spectral data using normalised training data, wherein the spectral data is normalised based on a different scaling to the normalisation of the training data.