Managed Event Tables for Database Telemetry Logging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for logging telemetry data from User-Defined Functions (UDFs) executing in secure environments are not scalable, as they require manual management of tables and explicit SQL queries, and the log data is often ad-hoc and unstructured, making it difficult to search and query.
Innovation Solution
The implementation of managed event tables (METs) that automatically collect and store telemetry data, including trace events, from UDFs executing in a secure sandbox environment, allowing for structured data capture and easy querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual management of tables and explicit SQL queries are used for logging telemetry data, then security control is improved, but scalability and ease of operation deteriorate
Solution Approach 1:
The system enables self-service logging where the telemetry logging mechanism automatically manages event tables and captures trace events without requiring manual SQL queries or table management by users. The logging system autonomously collects, structures, and stores telemetry data from UDF executions.
Solution Approach 2:
An intermediary logging system is introduced between the UDF execution environment and the storage layer. This intermediary automatically manages the complexity of table creation, data structuring, and storage operations, shielding users from manual management while maintaining security controls through the sandboxed execution environment.
2Ease of operation
If ad-hoc and unstructured log data is collected, then ease of collection is improved, but searchability and query efficiency deteriorate
Solution Approach 1:
The system transforms unstructured log data into structured telemetry events with defined parameters and schemas. Trace events are captured with standardized fields and attributes, converting the data from an unstructured state to a structured state that enables efficient searching, filtering, and querying while maintaining ease of collection through automated capture.
Solution Approach 2:
The logging system segments telemetry data into distinct, structured event types with specific attributes. Each trace event is divided into standardized components (event name, timestamp, attributes, etc.), making the data more searchable and queryable while maintaining systematic collection processes.
3Productivity
If automated collection and structured storage of telemetry data is implemented, then query efficiency and data usability are improved, but system complexity increases
Solution Approach 1:
The telemetry logging system is designed as a universal, multi-functional component that handles multiple operations (event capture, structuring, validation, and storage) through a single integrated mechanism. This universal logging framework serves various UDFs and telemetry sources without requiring separate complex systems for each function.
Solution Approach 2:
The structured storage system is self-configuring and self-managing, automatically creating appropriate table structures and data schemas based on the telemetry event types. This self-service capability reduces the need for manual system configuration and management complexity while maintaining high query efficiency through optimized data organization.
Data Source
AI summary
A system includes at least one hardware processor and at least one memory storing instructions that cause the at least one hardware processor to perform operations. The operations include configuring a processing stack in an execution node process. The processing stack includes a telemetry application programming interface (API). At least one configuration of a trace event is retrieved using an API call received by the execution node process. Telemetry information of the trace even is collected using the telemetry API based on the at least one configuration. An event table is updated based on the telemetry information.