Enriched Data Metrics Pipeline for Lower Network and Compute Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data metrics pipelines in computer systems often result in performance degradation due to the trade-off between data collection frequency and computing resources, leading to significant overhead and resource consumption.
Innovation Solution
Configuring an enriched data metrics pipeline using node configuration data from an unenriched pipeline, where data enrichers generate and store enriched data at the node level, reducing the need for frequent data requests to a management service.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data collection frequency is increased in an unenriched pipeline, then more data is obtained per unit time, but computing resources scale linearly and system performance degrades
Solution Approach 1:
The patent segments the data metrics pipeline into two distinct components: an unenriched pipeline that collects raw data at high frequency with minimal processing, and an enriched pipeline that performs aggregation and analysis at lower frequency. This segmentation allows each pipeline to operate independently with optimized resource allocation, resolving the contradiction between high data collection frequency and computing resource consumption.
Solution Approach 2:
The enriched pipeline performs preliminary aggregation and enrichment of data before it reaches the management service. By pre-processing data in the enriched pipeline, the system reduces the computational burden on the management service and avoids the need for high-frequency full-data processing, thereby reducing overall computing resource usage while maintaining productivity.
2Loss of energy
If an unenriched pipeline is used, then less computing resources are consumed per data collection, but more frequent collections are needed to obtain sufficient data
Solution Approach 1:
The patent divides the data processing workflow into two segmented pipelines: the unenriched pipeline handles frequent, low-cost data collection, while the enriched pipeline performs periodic aggregation and enrichment. This segmentation allows the system to maintain low per-collection resource consumption while reducing the total time to obtain sufficient processed data through parallel operation of both pipelines.
Solution Approach 2:
The unenriched pipeline operates continuously at high frequency to collect raw data, ensuring a steady stream of information without interruption. Simultaneously, the enriched pipeline continuously aggregates and enriches this data in the background. This continuous operation of both pipelines ensures that sufficient processed data is obtained over time without requiring frequent full-processing cycles, reducing both energy loss and time loss.
3Loss of information
If frequent data requests are made to the management service, then more comprehensive data metrics are obtained, but network and compute overhead increases
Solution Approach 1:
The enriched pipeline performs preliminary aggregation, filtering, and enrichment of data before it is made available to the management service. This pre-processing ensures that when the management service requests data, it receives already-processed, comprehensive metrics rather than raw data requiring further processing. This reduces the frequency and overhead of management service requests while maintaining data comprehensiveness.
Solution Approach 2:
The enriched pipeline acts as an intermediary between the unenriched data collection and the management service. It receives raw data from the unenriched pipeline, performs aggregation and enrichment operations, and then provides processed data to the management service. This intermediary layer reduces the computational and network overhead on the management service while ensuring comprehensive data metrics are obtained.
Data Source
AI summary
Techniques for configuring an enriched data metrics pipeline (DMP) include: obtaining node configuration data associated with an unenriched DMP for nodes of a heterogeneous computing platform, including (a) a first kind of node including an application programming interface (API) for obtaining unenriched data associated with the first kind of node and (b) a second kind of node including an API for obtaining unenriched data associated with the second kind of node; the unenriched DMP being configured to provide node-level unenriched data to a service according to a first schedule; the service being configured to generate node-level data metrics based on the unenriched data; based on the node configuration data, configuring an enriched DMP to provide node-level enriched data to the service according to a second schedule that is less frequent than the first schedule; the service being configured to generate fleet-level data metrics based on the node-level enriched data.