Distributed Data Validation Service for Error Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

As the volume and diversity of data collected for business metrics and analysis increase, manual and automated validation techniques struggle to effectively detect errors and ensure data integrity, leading to reduced efficacy in monitoring, reporting, and forecasting due to the complexity of validating large and diverse data sets.

Innovation Solution

A distributed data validation service is implemented, which configures analysis modules to validate data sets using defined rule sets, applies automated multi-dimensional table evaluations, detects cross-table data variance, and performs chronological failure detection, while also managing notification aggregation to identify and report errors.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual validation techniques are used to validate data sets, then validation accuracy can be maintained at a reasonable level, but validation efficiency and productivity deteriorate significantly as data volume increases

Engineering Contradiction:

Improvevalidation accuracyVSAvoidvalidation efficiency

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The validation service segments the validation process into multiple independent components including data ingestion modules, rule evaluation engines, and notification systems that can operate independently and in parallel, enabling scalable validation of large data sets while maintaining accuracy

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Manual validation processes are replaced with an automated distributed validation system that uses computational algorithms and rule-based evaluation engines to perform validation tasks, eliminating the need for manual intervention while maintaining or improving validation accuracy

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If automated validation techniques are implemented to improve validation efficiency, then productivity increases, but the system becomes unable to cope with complex validation scenarios involving large and diverse data sets

Engineering Contradiction:

Improvevalidation efficiencyVSAvoidcapability to handle complex validation scenarios

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The validation system employs dynamic rule evaluation where validation rules can be configured, modified, and extended without reconfiguring the entire system. The distributed architecture allows dynamic addition of new data sources and validation logic to handle evolving complex validation scenarios

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The validation service is designed as a universal platform that can validate multiple types of data from diverse sources including log data, metric data, and trace data using a common rule-based engine, making it adaptable to various validation scenarios without requiring specialized systems

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Loss of information

If the number and diversity of data sources increase to provide greater insight from data analysis, then the value and completeness of data collection improves, but the difficulty and complexity of validation increases

Engineering Contradiction:

Improvecompleteness of data collectionVSAvoidcomplexity of validation system

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The validation service acts as an intermediary layer between diverse data sources and validation rules, providing a standardized interface for data ingestion and rule evaluation. This abstraction layer simplifies the complexity by handling data normalization and rule management centrally, allowing multiple data sources to be validated without proportionally increasing system complexity

Inventive Principle:

Principle #24Intermediary (Mediator)

4Reliability

If manual validation is used to ensure data correctness, then data integrity can be maintained, but the system becomes ineffective for validating large data sets and the time required increases significantly

Engineering Contradiction:

Improvedata integrityVSAvoidvalidation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Manual validation processes are replaced with automated distributed validation services that use computational algorithms to validate large data sets rapidly, maintaining data integrity through consistent rule-based evaluation while reducing validation time from manual scales to automated computational scales

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The validation service operates continuously and automatically without manual intervention, performing validation operations on incoming data streams in real-time or near-real-time, eliminating the time loss associated with manual validation while ensuring continuous data integrity monitoring

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS10248508B1Distributed data validation service

Publication Date: 2019.04.02 AMAZON TECH INC

AI summary

A data validation service may validate data sets maintained for one or more data sources. Several rule sets may describe various rules used to validate one or more data sets. The rule sets may be automatically applied to respective data sets in order to validate the respective data sets according to a dynamically determined schedule for the application of the rule sets. Reporting events may be detected which correspond to a rule set. In response to detecting a reporting event, a responsive action may be performed as described in the rule set, such as providing notification of the reporting event.