Observability Data Merging for Long-Range Cluster Tracing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems struggle with storing and visualizing observability data across long time intervals due to volume constraints and storage inefficiencies, making it difficult to debug performance issues in node clusters.
Innovation Solution
A data management system retrieves and merges observability data sets from cloud storage, selecting and retaining relevant time-partitioned data blocks to generate a unified visualization of operations, enabling efficient storage and visualization of tracing data across user-specified time ranges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If observability data sets are stored in cloud storage environment, then storage capacity is improved, but data retrieval and merging complexity increases
Solution Approach 1:
The patent segments observability data into time-partitioned data blocks organized into multiple data sets stored in cloud storage. Each data set corresponds to a specific time interval and contains tracing data for a portion of the node cluster. This segmentation allows selective retrieval of only relevant data blocks needed for debugging, reducing the complexity of data retrieval and merging while maintaining large storage capacity.
2Loss of information
If time-partitioned data blocks are merged across multiple data sets, then visualization completeness is improved, but processing time increases
Solution Approach 1:
The patent extracts and merges only the specific time-partitioned data blocks that are relevant to the selected time range and node subset, rather than processing all data blocks in all data sets. The triage service identifies and retrieves only the necessary portions of observability data, completing the visualization without unnecessary processing overhead.
3Reliability
If all observability data is retained for long time ranges, then debugging capability is improved, but storage costs increase
Solution Approach 1:
The patent segments observability data into multiple time-based data sets stored in cloud storage, allowing the system to retain historical data for extended periods without incurring high continuous storage costs. When debugging is needed, only the specific time-range data sets required are retrieved and merged, providing full debugging capability for the selected period while minimizing actual storage resource usage at any given time.
Data Source
AI summary
A method for data processing is described. The method may include retrieving, from a cloud storage environment accessible to a data management system (DMS), two or more observability data sets associated with a set of time intervals within a selected time range. The observability data sets may contain a set of time-partitioned data blocks that include tracing data associated with operations performed by one or more nodes of a node cluster of the DMS during the selected time range. The method may further include merging the tracing data by selecting a first subset of overlapping time-partitioned data blocks, omitting a second subset of the overlapping time-partitioned data blocks, and retaining one or more non-overlapping time-partitioned data blocks. The method may further include transmitting data corresponding to a visualization of the merged tracing data.