Cloud Data Ingestion via Flexible Schema Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud computing monitoring services provide limited analytics and insights into machine/application performance, failing to meet the deep performance monitoring needs of cloud computing customers.
Innovation Solution
A data intake and query system that uses a flexible schema to process and store machine data from various sources, enabling field-searchable events with late-binding schema and pipelined search language for advanced query processing and data analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If cloud computing monitoring services use predefined analytics and metrics, then the system complexity is reduced and ease of operation is improved, but the measurement precision and depth of performance insights are limited
Solution Approach 1:
The system dynamically adapts its data collection and analysis capabilities based on user needs. The flexible schema allows the monitoring system to evolve from predefined metrics to custom, deep-performance analytics as users require more precise measurements, resolving the contradiction between ease of operation and measurement precision.
Solution Approach 2:
The system changes its operational parameters by allowing users to define custom metrics and analytics configurations. This enables the transition from simple predefined monitoring to complex, precision-oriented performance analysis without requiring a complete system replacement.
2Measurement precision
If cloud computing monitoring services provide comprehensive data collection and analysis, then measurement precision and performance insights are improved, but device complexity and system resource requirements increase
Solution Approach 1:
The monitoring service is designed as a universal platform that can handle both simple predefined metrics and complex custom analytics within a single system. The flexible schema and configurable architecture allow one system to serve multiple monitoring needs, reducing the complexity burden while maintaining high measurement precision.
3Reliability
If raw machine data is stored without structured schema, then data integrity and completeness are preserved, but data processing efficiency and query performance deteriorate
Solution Approach 1:
The system performs preliminary data structuring and schema validation during the data ingestion phase rather than during queries. This preliminary organization of raw machine data maintains reliability while enabling efficient processing and querying later, as the data is already prepared in an optimized format.
4Reliability
If cloud monitoring services use fixed analytics frameworks, then system stability and reliability are improved, but adaptability and versatility in meeting diverse customer needs are reduced
Solution Approach 1:
The monitoring service implements dynamic configurability where the analytics framework can adapt to different customer needs while maintaining core system stability. The flexible schema allows the system to evolve from fixed predefined analytics to customizable, versatile performance monitoring without sacrificing reliability.
Data Source
AI summary
In accordance with various embodiments of the present disclosure, a query for information related to machine data generated by one or more machine data sources of a cloud computing platform (CCP) is sent by a client computing device and to a cloud computing monitoring component of the CCP, where the query is formed using native query language of the CCP. As a result, the client computing device via a connector receives a first data object that is formatted in accordance with a first format associated with the CCP. The client computing device via the connector may then convert the first data object to one or more second data objects formatted in accordance with a second format that allows for enhanced ingestion by a data intake and query system.