Analytics Database Structuring Event Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing performance monitoring systems face challenges in efficiently storing and querying large volumes of event data from distributed computing systems, leading to high latency and storage overhead, while also lacking flexibility in handling unknown data schemas.
Innovation Solution
The implementation of an analytics database that structures event data through de-duplication and template reduction, storing data as structured event data and templates, allowing for fast ingestion, flexible querying, and reduced storage footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If event data is stored in raw format without structuring, then storage flexibility is maintained, but query latency increases and storage overhead increases
Solution Approach 1:
The patent segments event data into distinct fields and values, organizing raw data into structured format with separate columns for event type, timestamp, source, and other attributes. This segmentation enables efficient querying by allowing the system to directly access specific fields without parsing entire raw events, thereby reducing query latency while maintaining manageable complexity through systematic organization
Solution Approach 2:
The patent applies preliminary action by pre-structuring event data upon ingestion into a standardized format with consistent field names and types. This upfront structuring eliminates the need for complex query-time parsing operations, reducing query latency while the established schema maintains complexity at acceptable levels through reusable templates
2Quantity of substance
If all event data is stored in detail, then data completeness is maintained, but storage space consumption increases
Solution Approach 1:
The patent applies discarding and recovering by identifying and removing redundant information during data structuring. Common fields such as timestamps, event types, and source identifiers that appear across multiple events are standardized and referenced efficiently, discarding duplicate representations while recovering the essential information needed for complete analysis through the structured schema
Solution Approach 2:
The patent uses copying by creating standardized field definitions and value enumerations that represent repeated data patterns. Instead of storing identical information in multiple places, the system copies field schemas and value references, reducing storage space while maintaining data completeness through consistent referencing of the original data structures
3Adaptability or versatility
If the monitoring system handles unknown data schemas, then system flexibility is improved, but processing complexity increases
Solution Approach 1:
The patent applies dynamics by implementing a flexible schema registration system that can dynamically adapt to new event types and field structures. The monitoring system maintains a registry of known schemas that can be extended with new definitions, allowing the system to handle unknown data schemas by registering them dynamically without requiring complete system reconfiguration, thus improving flexibility while managing processing complexity through incremental adaptation
Solution Approach 2:
The patent implements universality through a standardized event schema framework that can accommodate multiple event types and data formats. The system uses universal field definitions and type mappings that work across different data sources and event structures, allowing the monitoring system to handle unknown schemas by mapping them to the universal framework, thereby improving adaptability while reducing processing complexity through consistent handling rules
Data Source
AI summary
A monitoring system is configured for de-duplicating data for storing in a database. The monitoring system is configured for receiving a message including a sequence of fields and corresponding field values; generating an entry in a first mapping table that associates each unique field and each unique field value of the message to a corresponding index value that is unique; generating a first vector including index values; ordering the index values of the vector, the ordered index values indicating a structure of the message; generating an entry in a second mapping table that associates the structure of the message with a structure index value; generating a second vector including the structure index value and an ordered set of index values representing the field values; and storing, in a database, the second vector.