Outlier Removal in NIR Spectroscopy Calibration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing near infrared (NIR) spectroscopy methods for predicting material properties struggle with accurately identifying and handling spectral outliers, which can impair the creation of reliable calibration functions, especially in complex mixtures like agricultural products, due to the lack of a rigid definition for outliers and subjective methods for detection.
Innovation Solution
A computer-implemented method that uses principal component analysis, singular value decomposition, and a dynamic threshold approach to automatically identify and remove spectral outliers from infrared spectra, ensuring a robust calibration model for predicting material properties.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If spectral outliers are included in the calibration dataset, then the calibration model may be more robust to natural variability, but the prediction accuracy deteriorates due to instrumental errors and measurement anomalies
Solution Approach 1:
The patent applies preliminary action by performing outlier detection and removal before building the calibration model. The method calculates distance measures for all spectra against the calibration mean, identifies outliers using statistical thresholds, and removes them prior to model construction. This prevents instrumental errors and measurement anomalies from degrading prediction accuracy while preserving natural variability through systematic preprocessing.
Solution Approach 2:
The patent extracts and removes spectral outliers from the dataset before calibration model development. By calculating distance measures and comparing against thresholds, the method identifies and extracts problematic spectra (those with excessive instrumental errors or anomalies) and removes them from the calibration set, thereby improving prediction accuracy while maintaining robustness to natural variability.
2Ease of operation
If subjective methods are used to identify outliers, then the process is easier to implement, but the reliability of outlier detection deteriorates due to lack of rigid definition
Solution Approach 1:
The patent transforms the subjective outlier identification process into an objective parameter-based method. It calculates distance measures (statistical parameters) for each spectrum against the calibration mean and compares these values against predetermined thresholds. This parameter-based approach provides a rigid, reproducible definition of outliers while maintaining ease of implementation through automated calculation.
Solution Approach 2:
The patent replaces the mechanical/manual subjective judgment process with an automated computational system. Instead of relying on operator intuition to identify outliers, the method uses algorithmic calculation of distance measures and automatic comparison against statistical thresholds, thereby eliminating subjectivity while maintaining operational simplicity.
3Adaptability or versatility
If all spectral data points are used for calibration, then the calibration model covers the full range of variability, but the precision deteriorates due to contamination by instrumental errors
Solution Approach 1:
The patent applies preliminary action by systematically preprocessing the spectral data to remove outliers before calibration model development. The method calculates distance measures for all spectra, identifies those exceeding statistical thresholds (indicating instrumental errors), and removes them prior to model building. This ensures the calibration model is trained only on high-quality data, improving precision while maintaining coverage of natural variability.
Solution Approach 2:
The patent extracts and removes contaminating spectral data points from the calibration dataset. By calculating distance measures and comparing against thresholds, the method identifies and extracts spectra affected by instrumental errors, removing them from the calibration set. This purification process improves calibration precision while preserving the full range of natural variability in the remaining data.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This method provides a reliable and efficient means to identify and exclude outliers, leading to improved accuracy and precision in predicting material properties using NIR spectroscopy, even in complex samples like feedstuffs and biological materials.
Implementation Method 1
Near infrared (NIR) spectroscopy is a useful tool for predicting a property value of interest of a material
Data Source
AI summary
The present invention relates to a computer-implemented method for predicting a property value of interest in a sample investigated by infrared spectroscopy. The method aims at generating a calibration function. To this end, a set of calibration samples is selected, whereby outliers are identified and removed from the set of calibration samples. Outliers are determined using principal component analysis and singular value decomposition. The threshold value separating outliers from the remaining samples is calculated on the basis of a predetermined formulae. The threshold value may also be increased stepwise in order to dynamically set the threshold value, which is preferable for spectroscopic devices not operated under laboratory conditions.