Cyber Event RowKey Indexing for Constant-Time Security Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional indexing schemas for cyber event data become overwhelmed by large volumes of records, leading to increased processing resources and storage costs, and are unable to efficiently manage and query petabytes of data without significant performance degradation.
Innovation Solution
A method for indexing cyber event data in a scalable database involves reformating data into a common intermediary format, generating unique hash values, and creating rowKey indexes for efficient storage and retrieval, allowing constant-time queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional indexing schemas are used to store cyber event data, then data can be stored and queried, but the system becomes overwhelmed by large volumes of records leading to increased processing resources and storage costs
Solution Approach 1:
The patent segments the indexing process by creating separate index structures (rowKey index, column index, hash index) that are independent from the main data storage. Each index handles specific query patterns, allowing the system to manage petabytes of data without requiring proportional increases in processing resources for all operations.
Solution Approach 2:
The patent introduces an intermediary indexing layer between the raw cyber event data and query operations. This intermediary structure reformats and reorganizes data into a common intermediary format with standardized attributes, enabling efficient queries without processing the entire raw dataset.
2Quantity of substance
If conventional indexing schemas are used, then data can be stored, but read speeds and querying performance degrade significantly as data volume increases
Solution Approach 1:
The patent performs preliminary indexing actions when data is ingested, creating multiple index structures (rowKey, column, hash) that are ready for immediate query operations. This preliminary organization eliminates the need to scan through entire datasets during queries, maintaining fast read speeds even as data volume grows to petabytes.
Solution Approach 2:
The patent adds multiple indexing dimensions beyond simple linear storage. By creating rowKey indexes, column indexes, and hash indexes simultaneously, the system enables queries to jump directly to relevant data sections across multiple dimensions, dramatically improving query performance without increasing data volume.
3Productivity
If processing resources are increased to handle large data volumes, then more data can be processed, but costs become prohibitively expensive
Solution Approach 1:
The patent creates copies of data in an optimized intermediary format with standardized attributes, separate from the original raw data storage. These copies are structured specifically for query operations, allowing the system to handle large data volumes efficiently without requiring proportional increases in processing resources for query operations.
Solution Approach 2:
The patent changes the data representation parameters by reformating cyber event data into a common intermediary format with standardized attributes. This parameter transformation enables more efficient storage and querying, improving processing capability while reducing the computational resources needed for data operations.
Data Source
AI summary
The present disclosure describes a method and system for processing and indexing cyber event data from a continuously updated distributed database. The method and system employ an indexing strategy mapping a unique rowKey for each cyber event to the serialized contents of the event. This indexing strategy enables constant-time queries to events provided query parameters consisting of one or more assets and optionally one or more timestamps.