Distributed Messaging System for Parallel Event Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies in handling large amounts of event information, requiring excessive time to process data, which is a challenge in providing timely services to a large number of users.
Innovation Solution
A distributed messaging system is employed, where a first server acquires event information, transmits a job creation request to a distributed message server, and a second server processes the information using at least one processor, then transmits the processed data to a front-end-related server, allowing for efficient and parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a large amount of event information is processed in a centralized manner, then data processing completeness is improved, but processing time increases proportionally to data throughput
Solution Approach 1:
The patent divides the centralized data processing system into multiple distributed server nodes that independently process different portions of event information. Each server can handle specific data subsets in parallel, transforming a single sequential processing bottleneck into multiple concurrent processing streams, thereby reducing total processing time while maintaining complete data handling capability
Solution Approach 2:
The patent transitions from a single-dimensional centralized processing architecture to a multi-dimensional distributed architecture across multiple servers. By adding the spatial dimension of distributed server deployment and the temporal dimension of parallel processing, the system achieves simultaneous processing of multiple data streams, fundamentally changing how data throughput scales with processing capacity
2Productivity
If more processors are allocated to handle increased data throughput, then data processing speed is improved, but system complexity increases
Solution Approach 1:
The patent designs distributed server nodes with standardized, universal architectures that can perform multiple functions: event information reception, parallel processing of different data subsets, and result aggregation. This multi-functionality allows the same hardware configuration to handle various processing loads without requiring specialized complex components for each processing task
Solution Approach 2:
The patent introduces a message queue system as an intermediary layer between event sources and processing servers. This mediator decouples the complexity of data distribution and task management from the processing logic itself, allowing processors to focus solely on data transformation while the message queue handles routing, load balancing, and coordination, thereby reducing overall system complexity
Data Source
AI summary
According to the present disclosure, a method of processing data in a data processing system using a distributed messaging system may include acquiring, by a first server, event information associated with at least one item, transmitting, by the first server, a job creation request to a distributed message server based on the acquired event information, acquiring, by a second server, a job request message generated in the distributed message server based on the job creation request, processing, by the second server, event information using at least one processor based on the job request message, and transmitting, by the second server, the processed event information to a front-end-related server.