Dynamic Edge Data Processing for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face delays and failed processes due to inherent delays in data transmissions and the use of non-current source data, which is incompatible in real-time, especially in high-volume data processing across multiple platforms, and are unable to dynamically process data based on network and CPU utilization changes.
Innovation Solution
A system and method for dynamic processing of temporal upstream and downstream data in communication networks, involving the construction of device clusters from edge computing nodes, extraction and transformation of metadata, and transmission of compatible data to downstream systems to reduce latency and adapt to dynamic network conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If downstream systems wait for upstream systems to provide current source data through mapping documents, then data accuracy is improved, but processing time increases and latency is worsened
Solution Approach 1:
The system performs preliminary actions by proactively pushing data change notifications from upstream systems to downstream systems before downstream systems need to request the data. Event producers detect changes in source data and immediately notify event consumers, eliminating the waiting time while ensuring data accuracy through real-time change propagation.
Solution Approach 2:
The system implements feedback mechanisms where downstream systems can subscribe to specific data changes and receive targeted notifications. The event-driven architecture creates a feedback loop where data changes are automatically detected and propagated only when relevant, reducing unnecessary processing time while maintaining data accuracy through selective real-time updates.
2Productivity
If conventional systems use prior source data that is not current, then processing speed is improved, but data compatibility and reliability deteriorate
Solution Approach 1:
The system maintains continuity of useful action by establishing continuous real-time data streams from upstream to downstream systems. Event producers continuously monitor source data changes and push updates immediately, ensuring downstream systems always process current data without interruption. This continuous flow maintains both processing speed and data compatibility simultaneously.
Solution Approach 2:
The event-driven architecture enables self-service where the system automatically detects data changes and propagates them without manual intervention. Event producers autonomously monitor source systems, detect changes, and push notifications to consumers, ensuring data currentness and compatibility while maintaining high processing speed through automated real-time updates.
3Productivity
If multiple edge nodes and applications are deployed across various platforms for high volume data processing, then processing capacity is improved, but system complexity increases
Solution Approach 1:
The system applies segmentation by dividing the distributed data processing architecture into independent event producers and event consumers that communicate through standardized event channels. Each edge node and application is segmented into discrete functional units that can be deployed across multiple platforms independently, reducing system complexity while maintaining high processing capacity through modular event-driven interactions.
Solution Approach 2:
The event-driven architecture provides universality by creating a platform-agnostic event subscription mechanism that works across diverse edge nodes and applications. The standardized event production and consumption interfaces enable different platforms to participate in the same data processing workflow, reducing complexity through unified communication protocols while maintaining high processing capacity across multiple platforms.
4Loss of time
If downstream systems request current modifications to source data, then data currency is improved, but network utilization and processing overhead increase
Solution Approach 1:
The system extracts and pushes only the specific data changes that are relevant to downstream systems, rather than transmitting entire data sets or responding to broad requests. Event producers extract only the modified data elements and push them selectively to subscribed consumers, improving data currency while minimizing network utilization by transmitting only essential change information.
Data Source
AI summary
Embodiments of the invention are directed to systems, methods, and computer program products for dynamic processing of temporal upstream data and downstream data in communication networks. The invention is configured for dynamic processing and cascading of instance data and configuration files from edge node devices of a distributed network for reduction of latency in data transmissions. The invention involves constructing a first device cluster comprising one or more first cluster edge computing nodes of a plurality of edge computing nodes. In response to receiving a first downstream request for determining a current modification to the first source instance data, embodiments of the invention involves extracting at least one cluster configuration file associated with the at least one cluster edge computing node associated with first edge computing node, and processing the first technology application therewith.