Asynchronous Batch Audit via Commutative Checksums
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Auditing data integrity across distributed data storage systems is challenging due to data transformations and transfers, which can introduce corruption, especially when large amounts of data are involved, making it difficult to detect errors in operations performed on data objects across different systems.
Innovation Solution
A system generates order-independent checksums (fingerprints) for log entries from two data storage systems, comparing operation counts, data sizes, and transformed data object IDs to determine if operations and data match, using commutative operations like XOR to ensure accuracy regardless of order, and triggering notifications or recovery processes if discrepancies are found.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data objects are transformed and transferred between distributed data storage systems, then data can be stored in alternative formats required by different systems, but data corruption may be introduced during the transformation process
Solution Approach 1:
The patent applies preliminary action by generating checksums for log entries before data transformation and transfer operations. These checksums are created in advance and stored as reference values, enabling later verification of data integrity after transformation. This allows the system to maintain data format compatibility across different storage systems while detecting any corruption that may occur during transformation processes.
2Reliability
If auditing is performed on large amounts of transformed data operations, then data corruption can be detected, but the complexity of auditing increases significantly
Solution Approach 1:
The patent extracts the essential verification information by generating compact checksums from log entries. Instead of auditing and comparing entire large datasets, the system extracts and compares only the checksum values, which are much smaller in size. This dramatically reduces the complexity of the auditing system while maintaining the ability to detect data corruption in large amounts of transformed data operations.
3Reliability
If traditional auditing methods are used to verify data integrity across distributed systems, then data corruption can be detected, but the process becomes computationally expensive and time-consuming
Solution Approach 1:
The patent creates simplified copies of data verification information in the form of checksums. Instead of performing computationally expensive comparisons of entire datasets, the system uses these checksum copies to quickly verify data integrity. This approach maintains accurate detection of data corruption while significantly reducing the time required for auditing operations across distributed data storage systems.
Data Source
AI summary
First audit information corresponding to a first set of log entries associated with a transformation performed on first data is obtained, where the first audit information includes a first commutative result produced by applying a commutative function to object identifiers associated with the first set of log entries. Second audit information corresponding to a second set of log entries associated with the transformation performed on second data is obtained, with the second data being a different representation of the first data and where the second audit information includes a second commutative result produced by applying the commutative function to object identifiers associated with the second set of log entries. The first commutative result is compared with the second commutative result to validate whether the second data matches the first data. One or more actions are performed depending on whether the second data is the successful transformation of the first data.