Event Context Enrichment via Optimized Row Key Structures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for event context enrichment in distributed computing systems are inefficient due to their general-purpose nature, requiring complex queries and being slow for indexing large volumes of machine data, making it difficult to determine ancestor and descendant processes and their metadata efficiently.
Innovation Solution
A database structure and system that includes a distributed storage system with specific table structures like the 'execs' and 'parents' tables, allowing for efficient processing of fetch, back-trace, and follow enrichment queries through optimized row key formats and operations, facilitating easier retrieval of process creation events and their relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a general-purpose query language is used to index and query machine data, then flexibility in querying is improved, but query complexity and computational cost increase significantly
Solution Approach 1:
The patent segments the monolithic general-purpose query system into specialized components: a process index for efficient process lookup, a parents table for ancestor relationships, and an execs table for execution metadata. This segmentation allows simple targeted queries for process enrichment while maintaining the option for complex queries when needed.
Solution Approach 2:
The patent introduces intermediate data structures (process index, parents table, execs table) that act as mediators between the raw machine data and the query interface. These intermediaries pre-organize data in optimized formats, eliminating the need for complex join operations and reducing query complexity while preserving flexibility.
2Ease of operation
If a general-purpose query system is used for event context enrichment, then ease of use is improved, but processing speed and efficiency deteriorate
Solution Approach 1:
The patent performs preliminary actions by pre-indexing process data, pre-computing parent-child relationships in the parents table, and pre-storing execution metadata in the execs table during data ingestion. This preliminary organization enables fast direct lookups during query execution without requiring complex runtime computations, thus improving processing speed while maintaining ease of use.
3Loss of information
If complex join operations are performed to retrieve process metadata, then completeness of information is improved, but computational cost and time increase
Solution Approach 1:
The patent merges related process information into integrated data structures: the process index combines process identifiers with metadata, the parents table merges ancestor relationships with process identifiers, and the execs table merges execution details with process information. This merging eliminates the need for multiple separate join operations while ensuring information completeness, thus reducing query execution time.
Data Source
AI summary
A database structure and a system that uses the structure to facilitate efficient context enrichment of low-level events occurring in a distributed computing system. In one aspect, the database structure comprises a table accessible to a distributed storage system. The table comprises a plurality of rows. Each row represents a corresponding process creation event of a particular process at a particular host at a particular time and assigned a particular event identifier. Each row comprises a row key identifying the particular host, the particular process, the particular time, and the particular event identifier of the process creation event corresponding to the row. The particular time and the particular event identifier are stored as part of the row key in a bitwise one's complement format. The row key structure facilitates efficient identification of a process creation event where only hostname and the process identifier of the process creation event are known.