Distributed Feed Processing System for High Volume Real-Time Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Downstream feed systems face challenges in processing high volumes of transactions due to technical resource constraints such as memory and CPU limitations, leading to potential overload and non-compliance with regulatory reporting times.
Innovation Solution
A method and system for dynamically distributing data processing across a network based on available technical resource capacity, prioritizing feed requests, and allocating them to suitable computing resources, both locally and remotely, to ensure efficient processing and compliance with service level agreements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If real-time processing of high volume transactions is performed at a single downstream feed system, then processing speed is improved, but technical resource constraints (memory, CPU, capacity) cause system overload
Solution Approach 1:
The patent divides the single downstream feed system into multiple distributed computing resources (feed workers) across different servers and locations. Each worker processes a subset of the feed requests, segmenting the overall processing load to prevent any single system from becoming overloaded while maintaining collective processing speed.
Solution Approach 2:
The patent transitions from a single-location processing model to a multi-dimensional distributed architecture where computing resources are spread across multiple servers, potentially in different geographic locations. This dimensional expansion allows the system to handle high volumes without concentrating resources at one point, avoiding overload.
2Productivity
If more technical resources are allocated to process high volume transactions, then processing capability is improved, but resource constraints (memory, CPU, capacity) limit available capacity
Solution Approach 1:
The patent creates a universal pool of computing resources where multiple feed workers can be dynamically allocated to process different types of feed requests. Each worker is designed to handle various processing tasks, making the resource pool versatile and able to adapt to different processing demands without requiring dedicated resources for each function.
Solution Approach 2:
The patent implements dynamic resource allocation where the system can adjust the number and distribution of active feed workers based on current processing demands. When transaction volumes increase, more workers can be activated; when volumes decrease, workers can be scaled back, allowing the system to optimize productivity while efficiently utilizing available technical resources.
3Device complexity
If feed requests are processed in strict sequential order, then resource allocation is simplified, but processing time increases due to inability to parallelize
Solution Approach 1:
The patent introduces a feed request queue and dispatcher as intermediary components between feed requestors and processing workers. The dispatcher manages the complexity of allocating requests to multiple workers, using the queue to coordinate access and ensure proper ordering when needed, while enabling parallel processing where possible. This intermediary layer handles the coordination complexity, allowing workers to process requests in parallel without direct complex interactions.
4Productivity
If distributed computing resources are used across the network, then processing capacity is improved, but system complexity and coordination overhead increase
Solution Approach 1:
The patent implements feedback mechanisms where feed workers report their status, processing progress, and resource availability back to the central dispatcher. This feedback loop allows the dispatcher to make informed decisions about request allocation, balancing the distributed system dynamically. The feedback mechanism simplifies coordination by providing real-time visibility into system state, enabling the dispatcher to optimize resource utilization without requiring complex predetermined coordination protocols.
Data Source
AI summary
A method for distributing data process according to technical resource capacity availability and constraints across a network is disclosed. The method includes receiving, at a database, a feed request for processing by a computing resource, among a network of computing resources that are geographically dispersed. Further, the method includes identifying, among the network of computing resources, a computing resource for processing the feed request based on the at least one attribute and processing capacity of the computing resources, and assigning the feed request for processing to the identified computing resource in real-time and without predetermined assignment to process increase in data volume by leveraging remotely located and/or underutilized computing resources.