Ordered Rules-Based Dataset Cleaning for Column Metadata Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in managing large numbers of datasets with inconsistencies across datasets, such as differing column names and data types, which hinder data correlation and transformation, requiring labor-intensive manual mapping that becomes unsustainable as dataset numbers increase.
Innovation Solution
A rules-based dataset cleaning system that automates the unification of semantics across datasets by defining ordered lists of functions and actions to synchronize column metadata, allowing for consistent and reproducible cleaning operations that can be applied to multiple datasets automatically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual mapping is used to unify column metadata across datasets, then data consistency can be achieved, but the time and labor required increases linearly with the number of datasets
Solution Approach 1:
The system enables self-service automated cleaning by allowing users to define cleaning rules once that then automatically apply to multiple datasets. The rules engine autonomously identifies and corrects metadata inconsistencies without requiring manual intervention for each dataset, making the system serve itself rather than requiring continuous human effort.
Solution Approach 2:
The system changes the parameter of data cleaning from manual operation to automated rule-based operation. By transforming cleaning rules into machine-executable parameters and algorithms, the system maintains data consistency while dramatically reducing the time investment required, as rules can be applied repeatedly without redefinition.
2Reliability
If manual cleaning operations are performed on each dataset, then individual expertise can be applied, but the process cannot be consistently reproduced across different datasets
Solution Approach 1:
The system performs preliminary action by requiring users to define cleaning rules in advance before actual data cleaning occurs. These pre-defined rules capture individual expertise and ensure consistent application across all datasets, eliminating the need to manually repeat the same cleaning logic for each dataset while maintaining reliability.
Solution Approach 2:
The cleaning rules are designed to be universal and multi-functional, capable of being applied across multiple datasets with different structures and purposes. A single rule set can consistently clean metadata across diverse datasets, making the system both reliable and easy to operate without requiring dataset-specific manual intervention.
3Quantity of substance
If the number of datasets increases, then more data becomes available for analysis, but the manual effort required to clean and maintain them becomes unsustainable
Solution Approach 1:
The system scales to handle increasing data volumes by enabling self-service automated cleaning. Once rules are defined, they autonomously process any number of datasets without requiring proportional increases in manual effort, allowing the organization to grow its data assets without hitting productivity ceilings.
Solution Approach 2:
The system implements periodic action by allowing cleaning rules to be applied repeatedly and on-demand to datasets as they are added or updated. This periodic application of the same rules ensures that as data volume grows, the cleaning process remains efficient and scalable without requiring continuous manual redefinition of cleaning logic.
Data Source
AI summary
One or more processors store rules for performing rules-based cleaning operations on a plurality of datasets, wherein each rule comprises one or more functions to be executed against a dataset during the rules-based cleaning operations, the one or more functions each having one or more associated conditions and actions, wherein the one or more actions are performed on the dataset responsive to the one or more associated conditions being satisfied. The one or more processors further apply the rules to each of the plurality of datasets to perform the rules-based cleaning operations. To apply the rules to a given dataset, the one or more processors identify an ordered list of the one or more functions to be executed with respect to the given dataset during the rules-based cleaning operations and determine, for each of the one or more functions, whether the given dataset satisfies one or more conditions associated with a respective function of the one or more functions. Responsive to the given dataset satisfying the one or more conditions associated with the respective function, the one or more processors perform, on the given dataset, one or more actions associated with the respective function and provide a derived dataset comprising at least one modification to the given dataset resulting from the one or more actions associated with the respective function.