Data Consistency Validation for Multi-Source Engineering Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing and integrating data from multiple sources in a data engineering system is complex due to issues like data quality, compatibility, privacy, collaboration, and potential conflicts, especially with longitudinal data, requiring tedious and expensive continuous monitoring and recalibration.
Innovation Solution
A computer-implemented method using policies and a version control system to ensure data consistency by validating changes against consistency rules, versioning data, and managing conflicts, with a graph-based database for efficient search and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data from multiple sources are integrated manually with continuous monitoring and validation, then data consistency can be maintained, but the process becomes tedious, expensive, and error-prone
Solution Approach 1:
The patent implements preliminary validation by establishing a policy framework and consistency rules before data integration occurs. The system pre-defines validation criteria, data quality standards, and conflict resolution policies that automatically apply during data integration, eliminating the need for continuous manual monitoring and validation efforts
Solution Approach 2:
The system enables self-service data integration by automating the validation and consistency checking processes. The policy validator component automatically validates incoming data against predefined policies, and the version controller automatically resolves conflicts based on established rules, reducing reliance on manual intervention and expert oversight
2Measurement precision
If manual monitoring and recalibration of longitudinal data are performed continuously, then data accuracy can be sustained, but the process becomes tedious and expensive
Solution Approach 1:
The system performs preliminary validation of longitudinal data by establishing consistency rules and policies before data integration. The policy validator component pre-checks data accuracy against defined criteria, and the version controller prepares conflict resolution strategies in advance, eliminating the need for continuous manual monitoring and recalibration
Solution Approach 2:
The system implements automated feedback mechanisms where the policy validator continuously monitors data quality and provides feedback to the version controller. When inconsistencies are detected, the system automatically triggers validation and repair processes, maintaining data accuracy through automated feedback loops rather than continuous manual intervention
3Reliability
If data changes are validated against consistency rules before application, then data consistency is ensured, but the validation process adds time and complexity
Solution Approach 1:
The system performs preliminary validation by establishing a comprehensive policy framework and consistency rules before data integration occurs. The policy validator component pre-compiles validation criteria and the version controller pre-establishes conflict resolution policies, enabling rapid automated validation during data updates without adding significant time delays
Solution Approach 2:
The patent replaces manual validation mechanics with automated computational validation. The policy validator component uses algorithmic validation against predefined policies, and the version controller uses automated conflict resolution algorithms, substituting slow manual validation processes with fast computational validation that maintains data consistency while preserving productivity
Data Source
Figure 1
Figure 2
AI summary
The present invention relates to a computer-implemented method (100) for providing a data consistency between a first data source (20) and at least a second data source (22) in a data engineering system (200), comprising the following steps: - Providing (102) first data (10) from at least a first data source (20) and / or at least a second data source (22); - Detecting (104), by an update detector component (50), a change information (8) for the first data (10); - Providing (106), by a version area component (59), a first version information (70) of the detected change information (8); - Validating (108), the change information (8) for the first data (10), by a policy validator component (54), wherein a policy information (55) containing at least a consistency rule is executed on a common information model (52) to decide about a consistent state of the first data (10) defining whether the change information (8) can be applied to the first data (10), - Updating (112), by the version controller component (58), the first version information (70) to a second version information (72) of the validated change information (8), when the consistent state of the change information (8) is confirmed, and in case, the consistent state of the change information (8) is not confirmed, perform a step (118) of repairing the detected data inconsistency of the change information (8), before re-validating the change information (8) and updating the first version information (70); - Supplying (114) the change information (8) with the version information (70) as a production projection (80) to at least a client engineering application (82).