Distributed Database Validation via Column-Level Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for validating data sets in distributed database systems are resource-intensive and slow, particularly due to row-by-row validation in column-based formats like parquet, which affects performance and system throughput.
Innovation Solution
A method that validates data sets in a distributed database system by sending targeted requests to individual computers to check specific columns against validation criteria, allowing for column-by-column validation and reducing the need for individual data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If row-by-row validation is performed on column-based data formats, then data validation can be carried out, but resource consumption increases and validation speed decreases
Solution Approach 1:
The validation process is segmented into column-level operations rather than row-level operations. Each column is validated independently by sending targeted requests to the computer storing that column, allowing parallel validation of multiple columns and eliminating the need to transfer entire rows for validation checks.
Solution Approach 2:
The validation logic is extracted from the central computer and distributed to the computers storing the actual data columns. This allows validation to be performed where the data resides, eliminating the need to transfer data across the network and reducing communication overhead.
2Reliability
If individual data entries are transferred for validation, then comprehensive validation can be performed, but system throughput decreases and response time increases
Solution Approach 1:
Each computer storing a column of data performs the validation of its own data independently when requested, without requiring data to be transferred to a central validation point. The computer validates its local data against the provided criteria and returns only the validation result, enabling self-service validation that maintains throughput.
3Quantity of substance
If data is validated in a distributed system with multiple computers, then validation can be performed on large data sets, but communication channel utilization increases
Solution Approach 1:
The invention extracts only the essential validation criteria from the central computer and sends them to the appropriate computers for validation. Only the validation results (pass/fail) are returned to the central computer, rather than transferring the actual data values. This minimizes communication channel utilization while maintaining the ability to validate large data sets distributed across multiple computers.
Data Source
AI summary
A method is for validating a data set in a distributed database system in which the data set is stored distributedly on computers of the distributed database system. The method includes sending a request to one of the computers by a central computer. The request has a command to check for a given criterion of a partial data set of the data set on the computer to which the request was sent.