Market Data Feed Partitioning for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current market data processing systems face challenges in efficiently filtering and partitioning large volumes of real-time market data, leading to increased latency and packet loss, especially during high-volume trading events, due to the inability to effectively manage unpredictable bandwidth demands and fragmented trading landscapes.
Innovation Solution
A market data processing device (MDPD) that maintains an output-feed profile specifying a ticker-symbol subset and a ticker-symbol-based feed-partitioning scheme, filters and partitions order-book updates, and transmits customized market-data output feeds across multiple channels, allowing for intelligent switching and gap-fill services, reducing data volume and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If market data is transmitted without filtering and partitioning, then complete data is provided to downstream devices, but data volume and network bandwidth consumption increase significantly
Solution Approach 1:
The patent divides the complete market data feed into multiple partitioned feeds based on ticker symbols. Each partition contains a subset of the original data, allowing downstream devices to receive only the specific partitions they need. This segmentation reduces the overall data volume transmitted while ensuring that required information is not lost, as devices can subscribe to relevant partitions.
Solution Approach 2:
The system extracts and filters out only the necessary ticker symbols from the complete market data feed based on downstream device subscriptions. By removing unnecessary data elements (unsubscribed ticker symbols), the system reduces data volume while maintaining completeness of the required information for each specific downstream device.
2Loss of time
If market data is processed in real-time without pre-filtering, then all data is available immediately, but processing latency increases during high-volume trading events
Solution Approach 1:
The system performs preliminary filtering and partitioning of market data before transmission. By pre-processing the data feed according to downstream device subscriptions and creating partitioned feeds in advance, the system reduces the processing burden during real-time transmission. This preliminary action ensures that only relevant data is transmitted, reducing latency during high-volume trading events while maintaining fast data transmission speed.
3Adaptability or versatility
If complete market data feeds are transmitted to all downstream devices, then all devices receive all ticker symbol updates, but network bandwidth consumption and processing overhead increase
Solution Approach 1:
The patent implements segmentation by creating multiple partitioned data feeds, each containing updates for a specific subset of ticker symbols. Downstream devices can subscribe to only the partitions they need, enabling customized data reception. This approach provides adaptability and versatility for different device needs while significantly reducing network bandwidth consumption compared to transmitting complete feeds to all devices.
4Productivity
If market data is filtered and partitioned according to specific user needs, then data transmission efficiency improves, but system complexity increases
Solution Approach 1:
The system uses segmentation to organize ticker symbols into manageable partitions, which simplifies the filtering and routing process. Instead of handling complete feeds individually, the system divides data into standardized partitions that can be efficiently managed and transmitted. This segmentation approach improves data transmission efficiency while keeping system complexity manageable through structured organization.
Solution Approach 2:
The patent implements a universal partitioning framework that can serve multiple downstream devices with different subscription requirements. The same partitioning scheme and filtering logic are applied universally across all data feeds, allowing the system to handle customized data needs for multiple devices using a single standardized approach. This universality improves transmission efficiency while avoiding the complexity of implementing separate processing logic for each device.
Data Source
AI summary
Presently disclosed are systems and methods for generating customized filtered-and-partitioned market-data feeds. In an embodiment, an output-feed profile is maintained in data storage at a market-data-processing device (MDPD). The output-feed profile specifies a subset of ticker symbols and a ticker-symbol-based feed-partitioning scheme. An input feed of order-book updates to ticker symbols is received at the MDPD from an upstream device. At the MDPD, a customized market-data output feed is generated according to the maintained output-feed profile at least in part by filtering the input feed down to the order-book updates to ticker symbols in the specified subset and partitioning the filtered feed according to the specified ticker-symbol-based feed-partitioning scheme. The customized market-data output feed is transmitted from the MDPD to a downstream device.