Cloud Data Lake Partition Hashing for Completeness Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for ensuring data completeness in cloud data lakes are inadequate, posing challenges in maintaining data integrity, which is crucial for analytics and machine learning models.
Innovation Solution
A metadata table is maintained by the data owner with partition hash values for each partition in the cloud data lake, allowing clients to verify data completeness by comparing calculated hash values with those from the metadata table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If cloud providers store data in centralized repositories, then data capacity and scalability are improved, but data integrity and completeness cannot be guaranteed
Solution Approach 1:
The data owner calculates and stores hash values of data partitions in advance before uploading data to the cloud. These pre-computed hash values are stored in a metadata table that the client can later use to verify data completeness without requiring the data owner's continued involvement.
Solution Approach 2:
A metadata table serving as an intermediary structure is introduced between the cloud storage system and the client. This metadata table contains partition information and pre-stored hash values, enabling the client to independently verify data integrity without direct trust in the cloud provider.
2Reliability
If digital signature schemes are used to ensure data correctness, then data authenticity is improved, but data completeness cannot be verified
Solution Approach 1:
The data storage structure is segmented into multiple partitions, each with its own hash value stored in the metadata table. This segmentation allows the client to verify the completeness of specific partitions independently, enabling detection of missing or tampered data portions without requiring verification of the entire dataset.
Solution Approach 2:
The patent replaces traditional mechanical verification methods (such as checking individual file presence) with cryptographic hash-based verification. By computing and comparing hash values, the system can efficiently verify data completeness with high confidence without physically examining each data element.
3Reliability
If clients verify data completeness by checking individual files, then data completeness verification is possible, but verification efficiency deteriorates
Solution Approach 1:
The patent merges multiple individual file verification operations into a single partition-level hash verification. By storing hash values that represent entire partitions in the metadata table, the client can verify the completeness of all files within a partition by comparing a single computed hash against the stored hash, dramatically improving verification efficiency.
Data Source
AI summary
Methods for performing integrity verification of data obtained from cloud data lake are provided. Aspects include transmitting a request for a data set to the cloud data lake, receiving a plurality of file names from the cloud data lake, and extracting, from the plurality of file names, one or more partitions of the cloud data lake that stores the plurality of files. Aspects also include calculating a partition hash value for each of the one or more partitions, obtaining a metadata table created by a data owner of the plurality of files, and obtaining verified partition hash values from the metadata table. Based on a determination that one or more of the verified partition hash values are not equal to corresponding calculated partition hash values, aspects include creating an alert that the plurality of files received from the data lake is not a complete response to the request.