Multiscale Regression Outlier Detection for Data Curation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing signal acquisition systems are contaminated with noise and outliers, leading to valuable information loss and distortion, and there is a need for a robust, scalable, and customizable outlier detection and curation tool that can handle diverse data sources effectively.
Innovation Solution
A method that flags outliers in received data, stores them, removes them, and uses contextual custom regression techniques for refinement, applicable in both time-series and matrix-like data structures, facilitating data curation on various scales and domains to produce filtered curated data and outlier data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data smoothing or rejection methods are used, then noise and outliers are removed, but valuable information is lost and good data points are distorted
Solution Approach 1:
The patent applies parameter changes by using multiscale regression models that operate at different scales to detect outliers. Instead of using fixed thresholding or simple smoothing parameters, the system adapts parameters dynamically based on the local data structure and scale, allowing precise outlier detection while preserving genuine data variations.
Solution Approach 2:
The patent segments the data analysis process into multiple scales and levels. By dividing the detection process into different scales (local vs. global patterns) and using hierarchical regression models, the system can identify outliers at appropriate scales without affecting data points at other scales, thus preserving valuable information while removing true outliers.
2Measurement precision
If a robust and highly customizable outlier detection tool is developed, then detection accuracy improves, but system complexity increases
Solution Approach 1:
The patent implements universality by creating a multiscale regression framework that can handle diverse data types and outlier patterns through a unified approach. The same core regression-based methodology works across different scales and data characteristics, reducing the need for multiple specialized algorithms while maintaining high detection accuracy.
Solution Approach 2:
The system employs dynamics by making the regression models adaptive and scalable. The multiscale approach allows the system to dynamically adjust the level of detail and computational intensity based on the data being analyzed, providing high precision when needed while maintaining operational simplicity through automated scale selection.
3Productivity
If real-time outlier detection is implemented, then data processing speed is maintained, but computational complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing regression models at multiple scales before actual outlier detection is needed. This preprocessing step creates a hierarchical structure of models that can be quickly applied to new data in real-time, avoiding the computational burden of building models from scratch during processing.
Solution Approach 2:
The computational task is segmented into hierarchical levels where coarse-scale models provide quick initial filtering and fine-scale models handle detailed analysis only where needed. This segmentation allows real-time processing by avoiding unnecessary computation at all scales for every data point, reducing overall computational complexity while maintaining speed.
Data Source
AI summary
A method includes receiving data, flagging outliers in the received data, each of the outliers representing extreme data values in the received data that stand out greatly from an overall pattern of data values at a given scale in the received data, storing the flagged outliers in an outlier file, removing the flagged outliers from the received data, curating the flagged data points with respect to various neighborhood kernel sizes, and further curating data points with small perturbations using contextual custom regression techniques as a second order refinement process.