Ontological Log Data Structure for Rapid Cross-Source Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Corporations face challenges in rapidly processing and analyzing large quantities of disparate log data from various sources, such as user access logs and VPN connections, which are scattered across multiple databases, making it difficult to identify specific computer systems, user activities, and application usage effectively.
Innovation Solution
A system that generates an ontological representation of objects from disparate log datasets, allowing for complex queries and dynamic user interfaces to efficiently ingest, process, and visualize log information, enabling users to quickly identify specific objects and events across multiple datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If disparate log datasets are stored across multiple databases, then data quantity and source diversity are improved, but data processing speed and analysis efficiency deteriorate
Solution Approach 1:
The patent merges disparate log datasets from multiple databases into a unified data structure with standardized schemas. This consolidation enables efficient querying and analysis across all log sources simultaneously, resolving the contradiction by maintaining data quantity while improving processing speed through unified access.
Solution Approach 2:
The patent creates a universal log data structure that can accommodate multiple types of logs (security authentication, user access, application usage) with a common schema. This multi-functional framework allows the system to process diverse log sources efficiently through a single processing pipeline.
2Loss of information
If complex queries are performed on large quantities of log data, then analysis depth and information completeness are improved, but processing time and computational resources deteriorate
Solution Approach 1:
The patent performs preliminary actions by pre-processing log data into standardized structures and pre-computing relevant attributes during data ingestion. This preparation enables complex queries to execute faster since the heavy lifting of data normalization and attribute extraction has already been completed.
Solution Approach 2:
The patent creates simplified copies of log data in standardized formats that preserve essential information while enabling efficient querying. These copied representations allow complex analyses to be performed on lightweight data structures rather than raw, unprocessed logs.
3Loss of time
If log data is rapidly ingested and processed, then information freshness and decision-making timeliness are improved, but data accuracy and error detection capability deteriorate
Solution Approach 1:
The patent incorporates feedback mechanisms that validate log data during ingestion and processing. The system checks for consistency, detects anomalies, and verifies data quality in real-time, ensuring accuracy is maintained even as processing speed increases.
Solution Approach 2:
The patent implements self-service error detection where the system automatically identifies and flags potential data quality issues without external intervention. This self-monitoring capability maintains data accuracy through continuous validation while preserving rapid processing throughput.
Data Source
AI summary
Systems and methods for rapid importation of data including temporally tracked object recognition. One of the methods includes receiving datasets each indicating information associated with one or more objects. Information indicating unique identifying information associated with the objects is accessed, and an updated dataset joining information from datasets that is associated with each object is generated. The updated dataset is maintained to include most recent versions of each of the datasets, with one or more datasets being replaced with more recent versions, and with one or more other datasets being propagated to be the most recent versions. Queries received from clients are responded to, with the queries indicating requests for specific information related to objects.