Deterministic Change Log Ordering for Distributed Transaction Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In validation architecture systems, indicia generated by nodes in a distributed system often mismatch due to indeterministic ordering of change log events, leading to erroneous transaction validation and potential database corruption.
Innovation Solution
The solution involves defining deterministic subunits of transactions, ordering these subunits based on computed indicia, and calculating deterministic indicia for each subunit to ensure consistent validation across nodes, using methods such as hash values and sorting techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If nodes in a distributed system process transactions independently and generate indicia locally, then processing speed and system throughput are improved, but indicia mismatches occur due to indeterministic ordering of change log events, leading to validation errors
Solution Approach 1:
The patent applies preliminary action by embedding deterministic ordering information (indicia) into change log events at the time of transaction processing, before validation occurs. Each node calculates indicia for its change log events in advance, using a deterministic algorithm that ensures identical input produces identical output across all nodes. This pre-computed indicia is stored with the change log events, enabling later validation without requiring complex coordination during the validation phase itself.
Solution Approach 2:
The patent changes the parameter of event ordering from indeterministic (based on physical arrival time or disk flush order) to deterministic (based on calculated indicia values). By transforming the ordering criterion from a physical property that varies across nodes to a computed property that can be replicated identically, the system ensures that all nodes arrive at the same event sequence independently, resolving the validation mismatch problem while maintaining parallel processing.
2Ease of operation
If change log events are ordered by physical arrival time or disk flush order, then processing simplicity is maintained, but indeterministic ordering causes indicia divergence across nodes, resulting in false validation failures
Solution Approach 1:
The patent introduces an intermediary element - the deterministic indicia calculation - that mediates between the physical change log events and the validation process. Instead of directly comparing raw change log events that may have different timestamps or flush orders across nodes, the system uses the calculated indicia as an intermediary representation that standardizes the events for comparison. This intermediary layer absorbs the variability in physical ordering while presenting a consistent view for validation.
3Reliability
If nodes validate transactions by comparing indicia in real-time, then database integrity is maintained, but complex validation logic and additional computational overhead are introduced
Solution Approach 1:
The patent extracts the complexity of deterministic ordering from the validation logic itself and embeds it in the change log event processing stage. By calculating and storing indicia values with each change log event, the system separates the ordering logic (embedded in indicia calculation) from the validation logic (simple comparison of pre-computed values). This extraction reduces validation complexity to a straightforward comparison operation while maintaining the sophisticated deterministic ordering requirement.
Data Source
AI summary
A method is provided for validating transactions in a system that includes indeterministic ordering of change log events by using deterministic indicia. At least some of the change log events include indeterministic values. The system includes a plurality of nodes. Each node in the system includes an instance of a database used by an application, and a change log associated with the instance of the database used by the application. Each node redundantly executes the same transactions to be validated. Each transaction includes one or more data manipulation language (DML) or data definition language (DDL) operations. Results of the DML or DDL, operations are stored in the change log as change log events, wherein the order of the change log events stored in the change log on one node is in a different order than the order of change log events stored in the change log of at least one of the other nodes. For each transaction executing at each node, indeterministic values of the change log events are pre-converted to deterministic values of the change log events. Deterministic subunits of the change log events are defined. Indicia are then computed for each subunit of the change log events. The subunits are ordered by the computed indicia. Deterministic indicia are then computed for the transaction from the ordered subunits. The transaction is then validated with at least one of the other nodes using the deterministic indicia. The validation indicates either a match or a mismatch of the deterministic indicia computed at the plurality of nodes.