Database Outlier Detection Using RANSAC and Data Descriptors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Maintaining data quality in databases is challenging due to the presence of 'dirty' data, such as duplicates and erroneous entries, which are difficult to identify without prior knowledge of the target object, especially when data is in formats like strings, integers, or bytea, and frequent user interactions complicate keeping a clean master copy.
Innovation Solution
The method involves extracting a subset of data, allocating data descriptors to create a model based on attributes like length, character set, co-occurrence, frequency, entropy, and similarity, using algorithms like RANSAC for outlier detection, and iteratively filtering the data set to identify and remove outliers, employing clustering processes to determine filtering thresholds and similarity scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data analysis requires prior knowledge and information about the target object to identify dirty data, then detection accuracy is improved, but ease of operation deteriorates because users must have specialized knowledge about data formats and characteristics
Solution Approach 1:
The system performs self-service by automatically analyzing data descriptors and generating outlier detection models without requiring user input about data characteristics. The algorithm autonomously extracts features from the data and identifies outliers, eliminating the need for users to possess specialized knowledge about data formats or detection methodologies.
Solution Approach 2:
Data descriptors serve as an intermediary between the raw data and the outlier detection algorithm. These descriptors automatically capture essential characteristics of the data (such as length, character set, co-occurrence, frequency, entropy, and similarity) and present them in a standardized format that the algorithm can process, bridging the gap between diverse data formats and the detection mechanism.
2Productivity
If multiple users constantly move data in and out of the database, then productivity is improved through frequent data access and modification, but data quality deteriorates due to the accumulation of dirty data
Solution Approach 1:
The system performs preliminary action by continuously and automatically detecting outliers in the background as data is accessed or modified. Rather than waiting for data quality to deteriorate significantly, the system proactively identifies and flags dirty data, allowing for timely intervention and maintaining data quality without restricting user productivity.
Solution Approach 2:
The system implements feedback by continuously monitoring data quality metrics and automatically updating the outlier detection model based on newly identified patterns. This closed-loop approach ensures that as more data is processed, the system becomes increasingly accurate at identifying dirty data, maintaining data quality while accommodating high user activity.
3Adaptability or versatility
If data is stored in formats like strings, integers, or bytea that are not easily recognized, then adaptability is improved to handle diverse data types, but difficulty of detecting and measuring deteriorates
Solution Approach 1:
The system applies parameter changes by transforming diverse data formats into a unified set of numerical descriptors. Instead of attempting to interpret and compare raw data in various formats directly, the system converts all data types into standardized parameters (length, frequency, entropy, etc.) that can be uniformly analyzed, making outlier detection feasible across heterogeneous data formats.
Solution Approach 2:
Data descriptors act as an intermediary layer between diverse data formats and the detection algorithm. These descriptors automatically extract and standardize characteristics from strings, integers, bytea, and other formats, presenting them in a consistent numerical format that the algorithm can process efficiently, thereby handling format diversity without increasing detection difficulty.
Data Source
AI summary
Various systems, methods, and processes for identifying outliers in a data set stored in a database are disclosed. A subset of data is extracted from a data set. Data descriptors are allocated to the subset of data. A model of the subset of data is created based on attributes of the data descriptors. An iteration of an outlier detection process based on the model is then executed. The outlier detection process evaluates the subset of data, and the outlier detection process evaluates the data set based on the results of the evaluation of the subset of data. The outlier detection process, which can implement and/or use a Random Sample Consensus (RANSAC) algorithm, identifies outliers in the data set stored in the database.