Distributed Worker Nodes for Real-Time Data Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data stream processing systems in distributed parallel computing, such as those using Spark and Storm, face challenges with scalability and efficiency due to strict programming requirements and resource-intensive management, leading to limited processing capacity and low efficiency in handling real-time data streams.
Innovation Solution
A data processing method and apparatus that utilizes multiple worker nodes across servers to process service processes in a distributed manner, allowing for flexible configuration and customization of service processes, enabling efficient processing of real-time data streams by invoking computing resources across servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If complex external control and management frameworks (such as Spark and Storm) are used to control the entire data stream computing process, then the data processing capability is improved, but the system complexity and resource consumption increase significantly
Solution Approach 1:
The patent extracts the core data processing functionality from complex external frameworks and implements it through simplified worker nodes that can be directly invoked by service processes. This eliminates the need for heavy management platforms while retaining essential processing capabilities.
Solution Approach 2:
Service processes directly invoke worker nodes to execute data processing tasks without requiring external control frameworks. The system achieves self-service by allowing services to autonomously manage their own computing resources through direct worker node invocation, reducing dependency on complex external management systems.
2Adaptability or versatility
If customized development is required for each service as a new data processing process, then the service functionality is improved, but the development time and resource investment increase
Solution Approach 1:
The patent segments data processing into independent worker nodes that can be selectively invoked by different services. Each worker node represents a reusable processing unit, allowing services to be customized by combining different worker nodes without requiring complete custom development for each service.
Solution Approach 2:
Worker nodes are designed as universal, reusable components that can be invoked by multiple different services. This multi-functionality allows a single worker node to serve multiple purposes across different service processes, reducing the need for customized development while maintaining service-specific functionality.
3Productivity
If a management platform for data stream computing is built with multiple component dependencies and environment dependencies, then the processing functionality is improved, but the scalability and deployment flexibility deteriorate
Solution Approach 1:
The patent extracts the essential data processing functionality from the management platform and implements it through standalone worker nodes. This eliminates the need for complex component dependencies and environment configurations, enabling easier scalability and deployment flexibility while retaining core processing capabilities.
4Productivity
If the management platform adjusts and optimizes system parameters extensively, then the processing efficiency is improved, but the resource consumption and time investment increase
Solution Approach 1:
Worker nodes are designed to execute processing tasks efficiently without requiring extensive parameter adjustment and optimization by the management platform. The simplified architecture allows worker nodes to operate effectively with minimal configuration, reducing the resources and time needed for system parameter optimization while maintaining high processing efficiency.
Data Source
AI summary
Embodiments of this application provide a data processing method, apparatus, and computer-readable storage medium. The method includes acquiring a data processing request of a target service, the request including data to be processed; invoking a first worker node of the first server to execute an operation corresponding to a first service process of the target service on the data to be processed to obtain an intermediate processing result, the first worker node being a worker node on the first server for processing the first service process; and transmitting the intermediate processing result to a second server according to a next service process in a plurality of service processes in the target service, so that the second server invokes a second worker node to execute an operation on the intermediate processing result, the second worker node being a worker node on the second server for processing the next service process.