Merging Downloaded Data with Real-Time Feeds via Sequence Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic trading systems face complexities and inefficiencies when merging downloaded data with real-time data feeds, leading to increased processing resources, delays, and inaccuracies due to race conditions and state management challenges.
Innovation Solution
A method that involves a client device comparing sequence numbers of downloaded data with real-time data to determine when the download is complete, allowing it to halt the download and consume only the real-time data feed, thereby minimizing caching and reconciliation steps and eliminating race conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If downloaded data is merged with real-time data feeds using existing systems, then data completeness is improved, but system complexity and processing resource consumption increase significantly
Solution Approach 1:
The patent uses sequence numbers as simplified copies or representations of data identity, allowing the system to track and merge downloaded data with real-time data feeds through sequence number comparison rather than complex data structure analysis. This reduces system complexity while maintaining data completeness.
Solution Approach 2:
The system performs preliminary actions by downloading historical data with sequence numbers before real-time data consumption. This allows the client to pre-process and store sequence number information, enabling efficient merging later without increasing real-time processing complexity.
2Reliability
If downloaded data is merged with real-time data feeds using existing systems, then data completeness is improved, but processing time and delays increase
Solution Approach 1:
By using sequence numbers as simplified data identifiers, the system avoids time-consuming complex data structure comparisons. The sequence number copying approach enables rapid matching and merging operations, significantly reducing processing time while ensuring data completeness.
Solution Approach 2:
The patent changes the parameter used for data identification from complex data structures to simple sequence numbers. This parameter transformation enables faster comparison and merging operations, reducing processing delays while maintaining accurate data integration.
3Measurement precision
If real-time data is cached during download to enable merging, then data accuracy is improved, but memory requirements and state management complexity increase
Solution Approach 1:
The patent extracts only the essential sequence number information from the full data structures for caching purposes. By taking out just the sequence numbers rather than caching entire data structures, the system maintains data accuracy for merging while dramatically reducing memory requirements and state management complexity.
Solution Approach 2:
The system uses lightweight sequence numbers as disposable identifiers for merging operations. These simple integer objects are much cheaper in terms of memory than full data structures, and can be created and discarded efficiently during the download and merge process without requiring long-term storage.
4Reliability
If multiple connections are used for downloading and real-time data reception, then data completeness is improved, but race conditions and timing problems increase
Solution Approach 1:
The patent copies sequence numbers from both downloaded data and real-time data feeds into a unified sequence number space. This allows the system to handle multiple connections and data sources while using simple sequence number comparisons to detect and resolve overlaps, eliminating complex race condition management requirements.
Solution Approach 2:
Sequence numbers serve as an intermediary mechanism that mediates between multiple data connections. By using sequence numbers as the common language for comparing downloaded and real-time data, the system can handle concurrent operations from multiple connections without complex synchronization and race condition management.
Data Source
AI summary
Methods and apparatuses for merging downloaded data with a real-time data are disclosed. An example includes receiving downloaded item of data from a download connection with a data repository and real-time item of data from a real-time data feed with a data publisher. In response to determining that the downloaded item of data includes the most recent downloaded item of data from the download connection, the downloaded item of data is stored as the last downloaded item of data. In response to determining that the real-time item of data is the most recent real-time item of data received from the real-time data feed, the real-time item of data is stored as the last real-time item of data from the real-time data feed. In response to determining a match between the last downloaded item of data and the last real-time item of data the download connection is ended.