Hierarchical Complex Filter Query for Data Quality

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing data quality assessment tools are limited in analyzing complex data quality criteria, as they can only utilize flat filters that check column values, failing to identify issues that require more intricate compositions or rule-based compliance.

Innovation Solution

Implementing complex filters, such as cleanse filters based on semantic meaning and rule filters based on pre-defined rules, which generate additional datasets for analysis, combined with hierarchical visualization to facilitate query generation and processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If flat filters are used to check column values, then the filter structure remains simple and easy to operate, but the data quality analysis capability is limited and cannot handle complex criteria

Engineering Contradiction:
Improvefilter operation simplicityVSAvoiddata quality analysis capability
Core Design Contradiction:
Ease of operationVSAdaptability or versatility

Solution Approach 1:

The patent segments the filter system into two distinct types: flat filters for simple column value checks and complex filters for advanced data quality criteria. This segmentation allows each filter type to specialize in its respective strength, maintaining operational simplicity for basic tasks while enabling sophisticated analysis when needed.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces a new dimension to the filter system by adding complex filters that operate beyond simple column value comparisons. This dimensional expansion enables the system to handle composite criteria, semantic meaning-based filtering, and rule-based compliance checks, thereby enhancing adaptability without compromising the existing flat filter functionality.

Inventive Principle:
Principle #17Another dimension (Dimensionality change)

2Adaptability or versatility

If multiple flat filters are applied additively, then more filtering criteria can be checked, but the analysis remains limited to column values and cannot identify complex data quality issues

Engineering Contradiction:
Improvefiltering criteria coverageVSAvoiddata quality assessment accuracy
Core Design Contradiction:
Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent creates a composite filtering system where complex filters combine multiple flat filters and logical operators (AND, OR, NOT) into unified filter expressions. This composite structure enables the system to assess complex data quality criteria such as 'total amount must be greater than sum of individual amounts' by integrating multiple column checks with logical reasoning, thereby improving measurement precision.

Inventive Principle:
Principle #40Composite materials

Solution Approach 2:

The patent introduces logical operators as intermediaries between flat filters and the final filtering result. These operators (AND, OR, NOT) act as mediators that combine multiple simple column value checks into complex logical expressions, enabling precise data quality assessment for composite criteria while maintaining the simplicity of individual flat filter operations.

Inventive Principle:
Principle #24Intermediary (Mediator)

3Measurement precision

If complex filters based on semantic meaning and rules are implemented, then comprehensive data quality analysis is enabled, but the filter system complexity increases

Engineering Contradiction:
Improvedata quality analysis granularityVSAvoidfilter system complexity
Core Design Contradiction:
Measurement precisionVSDevice complexity

Solution Approach 1:

The patent implements a dynamic filter system where the complexity of the filtering operation adapts to the needs of the analysis. Users can start with simple flat filters and progressively add complex filters with logical operators and semantic meaning checks only when required. This dynamic approach allows the system to maintain low complexity for simple tasks while providing access to advanced capabilities when needed, resolving the contradiction between analysis granularity and system complexity.

Inventive Principle:
Principle #15Dynamics

Data Source

PatentUS11080282B2Complex filter query of multiple data sets
Publication Date: 2021.08.03 SAP SE
  • US11080282B2 patent drawing
  • US11080282B2 patent drawing
  • US11080282B2 patent drawing

AI summary

In one embodiment, a complex query includes components that are arranged in a hierarchical structure including a first type of filter and a second type of filter and are connected by connectors. The method selects a first data set and selects a second data set for the components. The second data set being generated by processing data in the first data set for the second type of filter and the second data set includes entries describing a result of the processing. The first type of filter is applied to the first data set and the second type of filter to the second data set for the components where the information describing the result is used by the second type of filter to filter entries and first type of filter filters entries based on column values in the first data set. The method combines outputs of the components using the connectors.