In-Memory Parallel Search for Unstructured Data Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems are inefficient in handling unstructured data, requiring costly indexing and scanning that is limited in detecting embedded content, leading to undetected cybersecurity threats and inefficiencies in data management.
Innovation Solution
A system that processes data in volatile memory without indexing, using parallel processing and heuristics to analyze content and metadata, allowing real-time monitoring and automated actions on data as it is generated, and applying user-defined rules to determine which data to search or process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If indexing is used to process data, then search capability is improved, but cost and time consumption increase
Solution Approach 1:
The system performs preliminary extraction of content and metadata from compound files into volatile memory before processing. This allows the system to analyze data as it is generated rather than requiring post-generation indexing, thereby reducing time consumption while maintaining search capability through immediate availability of extracted information.
Solution Approach 2:
The system extracts content and metadata from compound files directly into volatile memory, separating the extraction process from the indexing process. This extraction approach enables rapid processing without requiring the creation and maintenance of indexes, thus reducing time and cost while preserving search functionality through the extracted data structures.
2Difficulty of detecting and measuring
If scanning is used to detect content, then detection capability is improved, but ability to examine embedded content is limited
Solution Approach 1:
The system segments compound files into their constituent components and extracts both content and metadata from each segment into volatile memory. This segmentation allows the system to thoroughly examine embedded content within compound files, overcoming the limitations of traditional scanning that treats compound files as monolithic units.
Solution Approach 2:
The system implements a nested extraction process where compound files are extracted to volatile memory, and their embedded content is subsequently extracted from those extracted files. This nested approach enables the system to penetrate multiple levels of file embedding and detect content within compound files, attachments, and nested structures.
3Measurement precision
If indexing is implemented, then search functionality is improved, but storage resources and complexity increase
Solution Approach 1:
The system extracts only the necessary content and metadata from compound files into volatile memory, eliminating the need for creating comprehensive indexes. This extraction approach reduces storage resource requirements and system complexity while maintaining search functionality through the extracted data structures that can be directly queried without index maintenance overhead.
Solution Approach 2:
The system creates copies of content and metadata in volatile memory for processing purposes, rather than maintaining permanent indexes on disk. This copying approach allows for rapid processing and search operations while reducing long-term storage requirements and system complexity, as the volatile memory copies can be discarded after processing.
4Productivity
If traditional data processing is used, then processing speed is maintained, but detection of cybersecurity threats is delayed
Solution Approach 1:
The system performs preliminary extraction and analysis of content and metadata as data is generated and stored in volatile memory, rather than processing data after it is fully written to disk. This preliminary action enables real-time detection of cybersecurity threats, reducing detection delay while maintaining processing speed through efficient in-memory operations.
Solution Approach 2:
The system continuously monitors and processes data as it is generated, maintaining uninterrupted analysis operations in volatile memory. This continuous processing approach eliminates detection delays associated with batch processing or post-generation indexing, while preserving high processing speeds through optimized in-memory operations that can quickly identify security threats.
Data Source
AI summary
This invention provides systems and methods for data processing by means of an ongoing background process on an end-user's computer. As a user receives and generates data, files are analyzed. A container file is opened into the volatile memory and its contents (including data and metadata) are extracted, without requiring an index to be created. The extracted components are analyzed based on predefined characteristics.