Event Log Management System for Schemaless Data Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current ETL solutions lack comprehensive and reusable methods to process schemaless data, which prevents the integration and analysis of event logs generated by heterogeneous applications due to the absence of a clear organizational structure and schema, making it difficult to load and query schemaless data in relational databases.
Innovation Solution
An event log management system infers an event-specific schema from common attributes within the schemaless data, partitions the data accordingly, and generates query-friendly datasets that can be processed by an ETL system, enabling the organization and loading of schemaless semi-structured data into relational databases for analytics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If schemaless data from heterogeneous applications is processed using current ETL solutions, then data integration is attempted, but the lack of organizational structure and schema prevents successful loading and querying in relational databases
Solution Approach 1:
The patent introduces an intermediary component that acts as a bridge between schemaless event log data and relational databases. This intermediary infers schemas from the unstructured data, creates appropriate organizational structures, and transforms the data into formats suitable for relational database storage and querying, thereby enabling integration without requiring changes to either the source heterogeneous applications or the target database systems
Solution Approach 2:
The system performs preliminary schema inference and data organization before loading data into the relational database. By analyzing event log data in advance, inferring appropriate schemas, and pre-organizing the data structure, the system prepares the data beforehand to ensure it meets relational database requirements, avoiding loading issues and enabling efficient subsequent querying
2Adaptability or versatility
If a generic schema is used to support schema variability among multiple applications, then flexibility is maintained, but the absence of event-specific organizational structure prevents effective data analysis
Solution Approach 1:
The patent segments the generic schemaless data into event-specific categories by inferring schemas for different types of events (e.g., user interactions, system events, errors). This segmentation organizes the previously undifferentiated data stream into distinct, analyzable groups while preserving the original flexibility to accommodate various application types, enabling targeted analysis of specific event categories
Solution Approach 2:
The system applies different organizational structures and schemas to different segments of the data based on their specific event types. Rather than imposing a single rigid schema, the patent creates locally optimized organizational structures for each event category, allowing each segment to maintain its unique characteristics while contributing to the overall organized dataset suitable for analysis
3Ease of manufacture
If schemaless data is stored without organizational structure, then data collection from heterogeneous sources is simplified, but data querying and analysis become difficult
Solution Approach 1:
The system performs preliminary organization and schema inference on the collected schemaless data before it needs to be queried or analyzed. By pre-processing the data to infer schemas and create organizational structures at the time of collection or storage, the system eliminates the need for complex querying operations later, maintaining simple data collection while enabling efficient subsequent analysis
Data Source
AI summary
In some implementations, a system may obtain, from a first data repository, a first dataset that includes event data associated with a generic schema. The system may infer an event-specific schema that defines an organizational structure for the event data based on common attributes identified among a plurality of events included in the event data using one or more data analytics functions. The system may store, in a second data repository, a second dataset in which the event data is partitioned based on the organizational structure defined by the event-specific schema. The system may generate a third dataset that includes a subset of the event data included in the second dataset that satisfies one or more registration parameters related to an extract, transform, load (ETL) use case. The system may provide the third dataset to an ETL system configured to process the third dataset based on the ETL use case.