Merkle Hash Tree Data Verification for Big Data Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for comparing large datasets in complex systems, such as power plants and medical devices, are inefficient due to resource and time constraints, and often miss errors in partial data comparisons or are not supported by all database systems.
Innovation Solution
The method involves generating Merkle hash trees for transformed data using ETL jobs and comparing their top hash values to determine data consistency, allowing for efficient detection of errors without transmitting large amounts of data, and can be integrated into the normal workflow without distorting the production system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complete data comparison is performed between source and target systems, then measurement precision is improved, but loss of time and use of energy worsen due to gigabytes and terabytes of data needing comparison
Solution Approach 1:
The patent extracts only the essential verification information from the complete dataset by computing Merkle hash tree root values. Instead of comparing all gigabytes or terabytes of data, the system extracts condensed hash representations that capture the essential integrity information, enabling rapid comparison without processing the full dataset.
Solution Approach 2:
The patent transforms the data comparison problem from comparing raw data values to comparing cryptographic hash values. By changing the parameter being compared from actual data content to Merkle root hashes, the system achieves both high precision (hash collisions are extremely rare) and high speed (comparing a few hash values is computationally trivial).
2Measurement precision
If complete data comparison is performed between source and target systems, then measurement precision is improved, but use of energy worsens due to resource constraints in Big Data testing
Solution Approach 1:
The system extracts only the Merkle root hash values from the complete datasets for comparison purposes. This extraction approach maintains full verification capability while consuming minimal computational resources, as computing and comparing a single hash value requires far less energy than processing entire datasets.
Solution Approach 2:
The patent creates cryptographic copies (hash values) of the original data that serve as efficient proxies for verification. These hash copies contain condensed information about the entire dataset's integrity, allowing verification without directly processing or transmitting the original large-volume data.
3Loss of time
If a percentage of data is compared between source and target systems, then loss of time and use of energy are reduced, but measurement precision worsens as errors in single-digit percentages are likely to be missed
Solution Approach 1:
The patent applies partial action by computing Merkle hash trees only for the specific datasets being tested rather than processing all possible data. However, within each dataset, the Merkle hash structure provides complete coverage verification - any single data point change alters the root hash, ensuring full precision despite the 'partial' nature of selective dataset testing.
Solution Approach 2:
The patent replaces the mechanical approach of sampling and statistically estimating data quality with a cryptographic verification system. The Merkle hash tree structure provides deterministic verification where the root hash mathematically guarantees the integrity of all underlying data points, eliminating the statistical uncertainty inherent in sampling methods.
4Measurement precision
If minus queries are used to compare database systems, then measurement precision is improved for detecting differences, but device complexity worsens due to requiring execution twice and support for minus queries
Solution Approach 1:
The Merkle hash tree approach provides universal compatibility across different database systems and platforms. Unlike minus queries which require specific SQL dialect support, the cryptographic hash-based verification method works uniformly across any system capable of computing hash functions, making the solution universally applicable without requiring special system features or dual-directional query execution.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to a method for automatically testing a process flow in which at least one operation (J1, J2, J3) is performed on input data (D1, D2, D3, D4), resulting in transformed data, comprising the following steps: - creating a first data structure (D1) based on the input data (D1, D2, D3, D4); - creating at least one further data structure (D2) based on the data transformed by the operation or on reference data; - comparing the at least two data structures using control values, wherein for the comparison - at least one first and one second basic control value (H01, H02, H11, H12) is formed for the first data structure and at least one further first and one further second basic control value (H01, H02, H11, H12) is formed for the further data structure (D2);- at least once each for the first data structure (D1) and the further data structure (D2), a control value (H0, H1, TH) superior to the basic control values (H01, H02, H11, H12) is formed from the first control value (H01, H02, H11, H12) and the second control value (H01, H02, H11, H12) or from the further first control value (H01, H02, H11, H12) and the further second control value (H01, H02, H11, H12), where a most recently formed, superior control value (H0, H1, TH) represents a top control value (TH); - a comparison of the top control value (TH) of the first data structure with the top control value (TH) of the further data structure (D2) takes place.