Edge-Core Analytics State Management for Partial Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed real-time edge-core analytics, existing systems face challenges in managing intermediate states during parameter updates across multiple edge nodes, leading to inconsistencies and errors due to asynchronous updates and network instability, where some nodes may operate with old or new parameters simultaneously.
Innovation Solution
An apparatus and method that manage state information for edge nodes, determining a mode based on the update status of each node to either reprocess streaming data with new parameters or continue analytics with existing data, ensuring data integrity by handling intermediate states through edge state management and reprocessing mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parameter updates are issued to multiple edge nodes simultaneously, then the system can achieve consistent analytics results, but network instability and large number of edges make it impractical to ensure all edges are updated at once, leading to intermediate states with mixed old and new parameters
Solution Approach 1:
The core side performs preliminary actions by issuing parameter updates to edge nodes and tracking their update status in advance. The system proactively identifies which edges have updated and which are still in intermediate states, allowing it to prepare reprocessing strategies before inconsistencies cause problems. This preliminary tracking and status monitoring enables the system to anticipate and handle mixed parameter states before they corrupt analytics results.
Solution Approach 2:
The system implements feedback mechanisms where edge nodes report their parameter update status back to the core side. The core side continuously monitors which edges have applied new parameters versus which are still using old parameters. This feedback loop allows the system to detect intermediate states and trigger appropriate reprocessing actions, ensuring analytics consistency despite asynchronous updates across the distributed edge network.
2Reliability
If the system reprocesses streaming data from all edge nodes with new parameters, then analytics consistency is maintained during partial updates, but processing time and computational resources increase
Solution Approach 1:
The system applies local quality by reprocessing only the specific data streams from edge nodes that are in intermediate states (those that have not yet updated to new parameters). Instead of universally reprocessing all data from all edges, the system identifies and targets only the local problematic areas where parameter mismatches exist. This selective reprocessing maintains analytics integrity while minimizing unnecessary processing overhead and time loss.
Solution Approach 2:
The system performs partial reprocessing action by conducting reprocessing only on data from edge nodes that require it, rather than reprocessing all data from all edge nodes. The core side determines the extent of reprocessing needed based on the actual update status of each edge node, applying just enough reprocessing to ensure consistency without the excessive action of reprocessing everything. This partial action approach balances reliability with processing efficiency.
3Productivity
If the system conducts analytics on streaming data without reprocessing, then real-time processing speed is maintained, but inconsistencies arise when some edge nodes have old parameters and others have new parameters
Solution Approach 1:
The core side acts as an intermediary between edge nodes and the analytics processing pipeline. It monitors parameter update status across edges and intervenes by triggering reprocessing only when intermediate states are detected. This intermediary role allows the system to maintain real-time processing speed during normal operation while selectively introducing reprocessing to correct consistency issues, thus mediating between the competing demands of speed and reliability.
Data Source
AI summary
Example implementations described herein are directed to the management of data received from an edge side when an update is issued to edge nodes from a core apparatus. When the edge nodes receive the update, the receipt of the update may not be uniform (e.g., due to latency, downtime, etc.), which results in an intermediate state where some edge nodes are updated and some edge nodes are not. Example implementations described herein address the processing of data from edge nodes when such an intermediate state occurs by conducting reprocessing of data when data is transmitted from edge nodes operating from an old configuration.