Event Stream Schema Versioning for Global Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

There is a need for techniques to update the schema and data processing format in a shared event stream in a time-based manner, ensuring reliable encoding and decoding of messages, while maintaining global consistency across applications.

Innovation Solution

An application connected to the event stream publishes a schema for data payloads, dynamically updates it, and publishes the revised schema, allowing other applications to apply the changes to subsequent messages, while also managing code updates by publishing version identifiers and updating instances accordingly.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If the schema is dynamically updated to reflect changes in data processing format, then the adaptability of the system is improved, but the reliability of message encoding and decoding may deteriorate due to version inconsistencies

Engineering Contradiction:

Improveschema update flexibilityVSAvoidmessage encoding reliability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent applies preliminary action by publishing schema update messages to the event stream before actual schema changes take effect. This allows receiving applications to prepare for and synchronize with upcoming schema changes, ensuring that both sending and receiving applications use compatible schemas for message encoding and decoding, thus maintaining reliability during transitions.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements feedback mechanisms where applications monitor the event stream for schema update messages and adjust their schema versions accordingly. This feedback loop ensures that all applications in the system remain synchronized with the current schema version, preventing encoding/decoding failures while allowing flexible schema evolution.

Inventive Principle:

Principle #23Feedback

2Adaptability or versatility

If multiple applications use different schema versions simultaneously, then the adaptability of individual applications is improved, but the global consistency of the event stream deteriorates

Engineering Contradiction:

Improveapplication version flexibilityVSAvoidglobal schema consistency

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The patent uses the event stream itself as an intermediary to propagate schema version information across all applications. By publishing schema update messages with version identifiers to the event stream, the system ensures that all applications receive and process schema change notifications in a standardized manner, maintaining global consistency while allowing individual applications to transition at their own pace.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary actions by notifying all applications of upcoming schema changes through the event stream before the changes take effect. This advance notification allows all applications to synchronize their schema versions, ensuring global consistency is maintained during transitions rather than allowing divergent versions to coexist.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If schema changes are implemented immediately without notification, then the productivity of the system is improved, but the loss of information about schema versions increases

Engineering Contradiction:

Improveschema update speedVSAvoidschema version tracking

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The patent maintains continuity of useful action by continuously publishing schema version information and update messages to the event stream. This continuous information flow ensures that schema version tracking is never interrupted, even as schemas evolve rapidly. Applications can continuously monitor the event stream to stay synchronized with the current schema version without information loss.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The system uses feedback mechanisms where schema update messages are published to the event stream, providing continuous information feedback to all applications about current schema versions. This feedback loop prevents information loss by ensuring that all applications have access to the latest schema version information, enabling them to adapt quickly while maintaining version awareness.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12175311B2Application code management using an event stream

Publication Date: 2024.12.24 IEX GROUP INC

AI summary

In a processing system, an identifier for a current version of code for an application is published in a message to a shared event stream. Any instances of the application within the processing system may update to the current version before processing other messages having sequence numbers after the message containing the identifier. In this manner, multiple instances of the application may be periodically updated to maintain global consistency of the application and the shared event stream.