Message Board Data Filtering for Low Latency Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data distribution methods, such as IP Multicast, introduce significant latency in financial markets due to processing overhead and physical distance, making it difficult to achieve real-time data delivery essential for programmatic trading applications.
Innovation Solution
A system and method for low latency data distribution using a message board that aggregates data from a dynamic system, filters it based on time predicates, and publishes filtered data to subscribers, leveraging cluster computing to migrate processing closer to the data source, thereby reducing latency and increasing throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If IP Multicast is used for data distribution, then wide area distribution capability is improved, but latency increases due to processing overhead and physical distance
Solution Approach 1:
The system segments the data distribution function by separating the data source from distribution nodes (message boards) that are geographically distributed closer to consumers. This segmentation allows data to be replicated and distributed from multiple nearby locations rather than originating from a single remote source, reducing latency while maintaining wide area coverage
Solution Approach 2:
Message boards act as intermediary nodes between the central data source and end consumers. These intermediaries receive data from the source and locally serve consumers in their geographic region, eliminating the need for data to traverse the entire network distance for every consumer and thus reducing latency
2Loss of information
If data is filtered through intermediate content inspection mechanisms, then data relevance to subscribers is improved, but latency increases due to processing overhead
Solution Approach 1:
Subscribers pre-define their filtering criteria (predicates) in advance and register them with message boards. The message boards compile and optimize these filter expressions beforehand, so when data arrives, the filtering can be executed efficiently using pre-compiled logic rather than ad-hoc processing, reducing filtering latency
Solution Approach 2:
The system creates optimized copies of filter expressions that are stored at the message board level. Instead of executing complex filtering logic at each consumer endpoint or through multiple intermediate inspection points, the filtered results are generated once at the message board using copied and cached filter logic, reducing overall processing time
3Device complexity
If processing is performed at remote consumers rather than near the data source, then data distribution simplicity is improved, but latency increases due to physical distance
Solution Approach 1:
The system adds a geographic dimension to the data distribution architecture by deploying message boards at multiple physical locations between the data source and consumers. This spatial distribution creates intermediate processing points closer to consumers, reducing the physical distance data must travel while maintaining architectural simplicity through standardized message board components
Data Source
AI summary
A method for low latency data comprises receiving a data stream comprising data from a data source, receiving the data via one or more queues, where each of the one or more queues is associated with one or more filters adapted to filter the data based at least upon a time predicate to create filtered data. Each of the one or more filters is established by one or more subscribers. The method also comprises receiving a portion of the filtered data that was filtered by a filter established by a particular one or more subscribers, and publishing the portion of the filtered data for the particular one or more subscribers.