Metrics Analysis Workflow for Machine Data Visualization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data generated by modern data centers and computing environments is challenging due to the vast variety and volume of data types and formats, often requiring flexible and efficient data processing and retrieval methods.
Innovation Solution
A data intake and query system that uses a flexible schema to process and store machine data as events with timestamps, allowing for field-searchable and minimally processed data storage, enabling late-binding schema application during search time for efficient keyword and field-based queries, and utilizing a pipelined search language for data processing and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If pre-processing is performed to extract and store only prespecified data items, then retrieval efficiency is improved, but data flexibility and analysis capability are reduced
Solution Approach 1:
The system performs preliminary indexing of all machine data at ingestion time, creating a searchable structure without discarding any data. This preliminary action enables both fast retrieval (by having data ready-indexed) and full flexibility (by retaining all original data for any future analysis needs).
Solution Approach 2:
The system changes the parameter of data retention from selective (only prespecified items) to comprehensive (all machine data). By storing complete events with all fields rather than extracted subsets, the system maintains adaptability while using efficient storage and retrieval mechanisms.
2Adaptability or versatility
If all machine data is stored minimally processed, then data flexibility and analysis capability are improved, but processing complexity and resource requirements increase
Solution Approach 1:
The system segments machine data into discrete events with structured fields during ingestion. Each event is parsed into standardized components (timestamp, host, source, field-value pairs), creating manageable units that can be efficiently stored, indexed, and queried without processing complexity during analysis.
Solution Approach 2:
The system replaces complex mechanical processing (filtering, transforming, aggregating raw data during analysis) with an information-based approach (structured indexing and metadata tagging). The complexity is shifted from analysis time to ingestion time, where data is organized into searchable fields and events that enable simple, efficient queries later.
3Quantity of substance
If prespecified data items are extracted and stored, then storage efficiency is improved, but loss of information occurs
Solution Approach 1:
The system creates a structured copy of all machine data in an indexed format during ingestion. Instead of storing only extracted fields, it preserves complete event data with all original fields while adding searchable index structures. This copying approach enables efficient storage through compression and indexing while maintaining full data completeness for any future analysis needs.
Data Source
AI summary
Embodiments are disclosed for a visualization system that facilitates exploration and analysis of large sets of data by providing for synchronized, independent visualizations of metrics or event-derived value sets. The visualizations can reflect a variety of data, including pre-prepared metrics, event-derived values, or a combination thereof. Global parameter controls can enable synchronized interaction with multiple visualizations, such as modifying parameters of respective visualizations with a single input. Local parameter controls can enable interaction with an individual visualization, independent of interaction with other visualizations. A variety of tools and interfaces are provided to manipulate the visualizations to facilitate analysis across a variety of metrics within a single interface.