Hierarchical Complex Filter Query for Data Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality assessment tools are limited in analyzing complex data quality criteria, as they can only utilize flat filters that check column values, failing to identify issues that require more intricate compositions or rule-based compliance.
Innovation Solution
Implementing complex filters, such as cleanse filters based on semantic meaning and rule filters based on pre-defined rules, which generate additional datasets for analysis, combined with hierarchical visualization to facilitate query generation and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If flat filters are used to check column values, then the filter structure remains simple and easy to operate, but the data quality analysis capability is limited and cannot handle complex criteria
Solution Approach 1:
The patent segments the filter system into two distinct types: flat filters for simple column value checks and complex filters for advanced data quality criteria. This segmentation allows each filter type to specialize in its respective strength, maintaining operational simplicity for basic tasks while enabling sophisticated analysis when needed.
Solution Approach 2:
The patent introduces a new dimension to the filter system by adding complex filters that operate beyond simple column value comparisons. This dimensional expansion enables the system to handle composite criteria, semantic meaning-based filtering, and rule-based compliance checks, thereby enhancing adaptability without compromising the existing flat filter functionality.
2Adaptability or versatility
If multiple flat filters are applied additively, then more filtering criteria can be checked, but the analysis remains limited to column values and cannot identify complex data quality issues
Solution Approach 1:
The patent creates a composite filtering system where complex filters combine multiple flat filters and logical operators (AND, OR, NOT) into unified filter expressions. This composite structure enables the system to assess complex data quality criteria such as 'total amount must be greater than sum of individual amounts' by integrating multiple column checks with logical reasoning, thereby improving measurement precision.
Solution Approach 2:
The patent introduces logical operators as intermediaries between flat filters and the final filtering result. These operators (AND, OR, NOT) act as mediators that combine multiple simple column value checks into complex logical expressions, enabling precise data quality assessment for composite criteria while maintaining the simplicity of individual flat filter operations.
3Measurement precision
If complex filters based on semantic meaning and rules are implemented, then comprehensive data quality analysis is enabled, but the filter system complexity increases
Solution Approach 1:
The patent implements a dynamic filter system where the complexity of the filtering operation adapts to the needs of the analysis. Users can start with simple flat filters and progressively add complex filters with logical operators and semantic meaning checks only when required. This dynamic approach allows the system to maintain low complexity for simple tasks while providing access to advanced capabilities when needed, resolving the contradiction between analysis granularity and system complexity.
Data Source
AI summary
In one embodiment, a complex query includes components that are arranged in a hierarchical structure including a first type of filter and a second type of filter and are connected by connectors. The method selects a first data set and selects a second data set for the components. The second data set being generated by processing data in the first data set for the second type of filter and the second data set includes entries describing a result of the processing. The first type of filter is applied to the first data set and the second type of filter to the second data set for the components where the information describing the result is used by the second type of filter to filter entries and first type of filter filters entries based on column values in the first data set. The method combines outputs of the components using the connectors.


