Time-Accurate Event Streams Without Consensus Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face challenges in achieving consistent global ordering of events in real-time with high throughput and low latency, as conventional solutions like consensus algorithms and proof-of-work methods result in low throughput and long latencies.
Innovation Solution
A distributed computing network architecture that separates roles into writers and readers, with writers globally time-synchronized to timestamp and digitally sign messages, and readers to authenticate and sequence these messages, ensuring consistent event streams across the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If consensus algorithms like Paxos and Raft are used to ensure agreement on event ordering, then reliability is improved, but productivity deteriorates due to heavy coordination overhead
Solution Approach 1:
The system segments the distributed computing network into specialized nodes with distinct roles: ordering nodes that determine event sequence, processing nodes that execute events, and storage nodes that maintain state. This segmentation eliminates the need for all nodes to coordinate through consensus algorithms, as only ordering nodes perform sequencing while other nodes operate independently, thereby maintaining reliability through role specialization while improving throughput by removing coordination overhead from the critical path.
Solution Approach 2:
The patent introduces an event stream as an intermediary data structure that carries event information across the distributed network. This event stream serves as a shared medium that all nodes can consume without requiring direct coordination or consensus. The event stream encapsulates event data, metadata, and ordering information, allowing nodes to process events independently while maintaining consistent views of the system state, thus improving productivity while preserving reliability.
2Reliability
If proof-of-work algorithms are used to achieve agreement on event ordering, then reliability is improved, but productivity deteriorates due to long latencies
Solution Approach 1:
The system performs preliminary ordering of events at the source or near-source nodes before events need to be processed across the network. Ordering nodes determine the sequence of events in advance and embed this ordering information in the event stream. This preliminary action eliminates the need for time-consuming proof-of-work consensus when events are processed, as the ordering is already established. Events can be processed immediately upon receipt, significantly reducing latency while maintaining reliable ordering through the pre-established sequence.
3Productivity
If multiple nodes process different instructions simultaneously, then productivity is improved, but reliability deteriorates due to disagreements on event order
Solution Approach 1:
Different nodes in the network are assigned different local qualities or responsibilities: ordering nodes specialize in determining event sequence, processing nodes specialize in executing events, and storage nodes specialize in maintaining state. Each node operates with high reliability within its specialized function without needing to coordinate with other nodes for the same function. This local quality specialization allows parallel processing across nodes while maintaining global ordering consistency, as each node type performs its designated task independently based on the event stream.
Data Source
AI summary
Embodiments of the present invention may provide an improved distributed computing system. Entities in the distributed computing system may be divided into four categories: writers, readers, gateways, and applications. End users may interact with the system via the applications through the gateways. The role of writers and readers may be separated to distribute computational burdens. Writers may generate messages for an event stream. The messages may include a timestamp for consistent global ordering. The readers may arrange messages from various writers based on the timestamps to generate globally time-consistent event streams.