Message Streaming for Event-Driven User Account Data Retrieval

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data management systems for online services face inefficiencies in computing resource usage and timeliness due to unnecessary API calls and batch processing, leading to system lag, increased operational costs, and environmental impact.

Innovation Solution

Implementing an Event-Oriented Architecture (EOA) using a message streaming system like APACHE KAFKA to manage API calls based on event triggers, reducing the need for real-time updates and batch processing by publishing messages with topic identifiers, allowing data aggregation and analysis only when changes occur.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If real-time API calls are made to retrieve data for every user request, then data freshness and responsiveness are improved, but system load and computational waste increase significantly

Engineering Contradiction:

Improvedata retrieval speedVSAvoidsystem efficiency

Core Design Contradiction:

SpeedVSProductivity

Solution Approach 1:

The system performs data retrieval in advance by publishing messages to a stream when data changes occur, rather than waiting for user requests. This preliminary action stores data in an immutable log, so when users request data, it is already available without requiring real-time API calls, thus improving speed while reducing system load.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

An intermediary message stream (Apache Kafka) is introduced between the data source and users. Instead of users directly triggering API calls to data sources, they subscribe to the message stream which delivers data changes asynchronously. This intermediary decouples the request-response cycle, improving responsiveness while reducing computational waste.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If batch processing is used to reduce system load, then computational waste is reduced, but data freshness and responsiveness deteriorate

Engineering Contradiction:

Improvesystem efficiencyVSAvoiddata update delay

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system uses periodic action in the form of event-driven message publishing. Whenever a data change event occurs, a message is immediately published to the stream. This creates a periodic update mechanism that is neither continuous (which would increase load) nor batched (which would cause delays), thus maintaining data freshness while improving system efficiency.

Inventive Principle:

Principle #19Periodic action

3Reliability

If unnecessary API calls are made during peak usage times, then data availability is maintained, but system load and operational costs increase

Engineering Contradiction:

Improvedata availabilityVSAvoidoperational cost

Core Design Contradiction:

ReliabilityVSUse of energy by stationary object

Solution Approach 1:

The message stream system operates autonomously, maintaining data availability without requiring active API calls during peak usage. Data is published to the immutable log when changes occur, and users can retrieve it on-demand from the stream. This self-service mechanism ensures data availability while eliminating unnecessary API calls that would increase operational costs during high-traffic periods.

Inventive Principle:

Principle #25Self-service

4Speed

If real-time data retrieval is implemented for all users, then responsiveness is improved, but computational waste and environmental impact increase

Engineering Contradiction:

ImproveresponsivenessVSAvoidcomputational waste

Core Design Contradiction:

SpeedVSLoss of energy

Solution Approach 1:

The system extracts only the necessary data changes and publishes them as messages to the stream, rather than retrieving and processing all data for every user. This extraction approach delivers only relevant updates to subscribed users, improving responsiveness while minimizing computational waste by avoiding unnecessary data processing.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS20250390363A1Systems and methods for message-based data retrieval

Publication Date: 2025.12.25 WELLS FARGO BANK NA

AI summary

A method may include detecting a scheduled transaction change for a future event with respect to a user account, the user account associated with a user account identifier; generating an electronic message according to a schema of a message streaming system, the electronic message include a topic identifier and the user account identifier; publishing the electronic message to the message streaming system; processing the electronic message from the message streaming system, wherein processing includes: parsing a payload of the electronic message to retrieve the user account identifier; accessing transaction details of the scheduled transaction change for the future event; and storing the transaction details in a staging database; in response to receiving a request to update an analysis of the user account, generating an analysis of a future state of the user account based in part on the data in the staging database; and storing the analysis.