Read-Based Data Service Distribution System for Legacy Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data retrieval and processing methods from data sources often result in wasteful bandwidth and processing resource usage, as data consumers constantly poll for updates even when data has not changed, particularly in cases where data sources are outside their control or part of legacy systems lacking eventing functionality.
Innovation Solution
A read-based data service distribution system periodically polls data sources, determines changes by comparing current and previous data using hashing, and publishes events to event streams only when changes occur, optimizing bandwidth and processing by reducing unnecessary data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data consumers constantly poll data sources for updates, then data consumers can obtain current data, but bandwidth and processing resources are wasted when data has not changed
Solution Approach 1:
The patent extracts only the changed data portions from the data source and publishes them as separate events to event streams, rather than transferring complete data sets. This allows data consumers to receive only what has changed, reducing bandwidth consumption and processing overhead while maintaining data currency.
Solution Approach 2:
The system implements a feedback mechanism where data consumers subscribe to event streams and receive notifications when changes occur. This feedback loop enables the system to respond dynamically to actual data changes rather than following a rigid polling schedule, optimizing resource usage while ensuring data consumers receive updates only when necessary.
2Reliability
If data consumers poll data sources periodically, then current data can be obtained, but processing time is wasted on unchanged data
Solution Approach 1:
The system extracts and publishes only the changed portions of data as events, allowing data consumers to process only relevant updates rather than complete data sets. This extraction approach significantly reduces processing time while maintaining data currency through selective event publication.
Solution Approach 2:
The system uses periodic polling of data sources to detect changes, but combines this with event-driven publication of only changed data. This hybrid approach maintains the simplicity of periodic checking while eliminating the waste of processing unchanged data, as events are published only when changes are detected.
3Loss of information
If complete data sets are transferred during polling, then data consumers receive all information, but bandwidth is wasted transmitting unchanged data
Solution Approach 1:
The patent extracts only the changed data portions from complete data sets and publishes them as events. This extraction mechanism ensures data consumers receive all necessary information about changes while minimizing bandwidth consumption by excluding unchanged data from transmission.
Solution Approach 2:
The system segments the data transfer process into change detection and event publication phases. By segmenting the data flow to separate changed portions from unchanged portions, the system maintains information completeness for relevant updates while reducing overall bandwidth consumption through selective transmission.
Data Source
AI summary
An apparatus comprises a processing device configured to perform a polling operation for at least one data source to obtain information for one or more designated topics of interest and to determine whether the information for a given one of the one or more designated topics of interest obtained during the polling operation comprises changed information. Responsive to determining that the information for the given designated topic of interest obtained during the polling operation comprises changed information, the at least one processing device is configured to generate one or more events comprising the changed information. The at least one processing device is further configured to publish the generated one or more events to one or more event streams, the one or more event streams being subscribed to by one or more data consumers.