Multi-Level Indexing for Machine Generated Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information retrieval methods are inadequate for efficiently indexing and searching large amounts of machine-generated data from disparate sources, particularly in the context of cyber security, where rapid analysis and correlation of security-related data are critical to detect and prevent unauthorized access and attacks.
Innovation Solution
A system and method that involves receiving data into a multi-level storage system, performing high-speed searching, nested searching, and distributed searching to identify and display records of interest in a time-sequenced manner, using plural index databases and index trees, and allowing flexible indexing and management of data fields, enabling efficient storage, retrieval, and correlation of security data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional information retrieval algorithms are used for indexing and searching document text, then text-based document searching is effective, but machine generated data with timestamp and specific fields cannot be efficiently processed
Solution Approach 1:
The patent divides the indexing and searching system into multiple specialized components: a timestamp-based indexing structure for time-range queries, a field-based indexing structure for specific data fields (username, IP address, security actions), and a multi-level searching architecture. This segmentation allows each component to optimize for its specific data type while maintaining overall system efficiency.
2Quantity of substance
If large amounts of data from disparate sources are collected and stored, then comprehensive security analysis is enabled, but search and retrieval time increases significantly
Solution Approach 1:
The patent implements preliminary indexing of data by timestamp and fields when data is first received and stored. The timestamp-based index organizes data chronologically, and field-based indexes pre-process specific fields (username, IP address, security actions). This preliminary organization enables rapid retrieval without scanning entire data sets during search operations.
Solution Approach 2:
The patent introduces index structures as intermediary layers between the raw data storage and the search/query interface. These indexes act as mediators that translate user queries into efficient data retrieval operations, avoiding direct scanning of large volumes of raw security data from disparate sources.
3Ease of operation
If detailed indexing on multiple fields is performed to enable flexible searching, then search capability is enhanced, but system complexity increases
Solution Approach 1:
The patent implements a universal indexing framework that handles multiple data types and fields through consistent timestamp-based and field-based indexing structures. This multi-functional approach allows the same indexing mechanisms to serve various search needs (time-range queries, field-specific searches, cross-field correlations) without requiring separate complex systems for each function.
Data Source
AI summary
A computer-implemented method of searching large amounts of machine generated data collected from disparate sources, comprises steps of: receiving data from the disparate sources into a multi-level system of storage blocks; high-speed searching at a broad level for groups of the storage blocks containing records of interest; nested searching at an intermediate level to determine specific storage blocks containing the records of interest, within the groups of the storage blocks; distributed searching at a fine level to identify the records of interest, within the specific storage blocks; and displaying the records of interest to a user. In a computer data storage device, a data structure comprises: plural data files each holding a data group from a large data set, wherein the data groups are permitted to contain overlapping data; and plural index trees within each data file, each index tree representing a key into a data group; wherein an absence of a key in one data file results in an absence of a corresponding index tree in the one data file. A computer storage medium has stored thereon instructions for a computer to perform the method described above.